abstract: In sparse linear bandits, a learning agent sequentially selects an action and receive reward feedback, and the reward function depends linearly on a few coordinates of the covariates of the actions. This has applications in many real-world sequential decision making problems. In this paper, we propose a simple and computationally efficient sparse linear estimation method called $\textsc{PopArt}$ that enjoys a tighter $\ell_1$ recovery guarantee compared to Lasso (Tibshirani, 1996) in many problems. Our bound naturally motivates an experimental design criterion that is convex and thus computationally efficient to solve. Based on our novel estimator and design criterion, we derive sparse linear bandit algorithms that enjoy improved regret upper bounds upon the state of the art (Hao et al., 2020), especially w.r.t. the geometry of the given action set. Finally, we prove a matching lower bound for sparse linear bandits in the data-poor regime, which closes the gap between upper and lower bounds in prior work.
author: Kyoungseok Jang
University of Arizona
`email@example.com`
Chicheng Zhang
University of Arizona
`email@example.com `
Kwang-Sung Jun
University of Arizona
`email@example.com`
bibliography: library-shared.bib
title: PopArt: Efficient Sparse Regression and Experimental Design for Optimal Sparse Linear Bandits

```latex
\begin{bibunit}[plainnat]

\setlength{\abovedisplayskip}{5pt}
\setlength{\belowdisplayskip}{5pt}
\setlength{\abovedisplayshortskip}{5pt}
\setlength{\belowdisplayshortskip}{5pt}

%-- for reducing space after algorithm environment

\maketitle

% \vspace{-10pt}
\begin{abstract}
% \vspace{-10pt}
In sparse linear bandits, a learning agent sequentially selects an action and receive reward feedback, and the reward function depends linearly on a few coordinates of the covariates of the actions.
This has applications in many real-world sequential decision making problems.
%, such as online advertising and personalized medicine.
In this paper, we propose a simple and computationally efficient sparse linear estimation method called \ensuremath{\textsc{PopArt}} that enjoys a tighter $\ell_1$ recovery guarantee compared to Lasso (Tibshirani, 1996) in many problems.
Our bound naturally motivates an experimental design criterion that is convex and thus computationally efficient to solve.
% and a new experimental design criterion that promotes tighter estimation guarantees compared to Lasso.
% our bound naturally motivates an experimental design criterion that is convex and thus computationally efficient to solve.
Based on our novel estimator and design criterion, we derive sparse linear bandit algorithms that enjoy improved regret upper bounds upon the state of the art (Hao et al., 2020), especially w.r.t. the geometry of the given action set.
% for data-poor regime and data-rich regime, respectively,
Finally, we prove a matching lower bound for sparse linear bandits in the data-poor regime, which closes the gap between upper and lower bounds in prior work.
% showing that our sparse linear bandit algorithm is optimal.
% close the gap between upper and lower bounds
\end{abstract}
%\chicheng{sometimes tighter?} \kj{I changed the wording.}
% \kj{would-be-great:
% \begin{itemize}
% \item (optional) maybe no need to explain what a sparse linear bandit is
% \item mention explicit regret bounds. $\tilde O( (H^2 sn)^{2/3})$ where $H^2$ is our novel quantity that captures the geometry of the action set. (also needs to explain $s$ and $n$).
% \end{itemize}

% }

\textfloatsep=.6em

% \vspace{-20pt}
\section{Introduction}
\label{sec:intro}
% \vspace{-8pt}

In many modern science and engineering applications, high-dimensional data naturally emerges, where the number of features significantly outnumber the number of samples.
In gene microarray analysis for cancer prediction~\cite{ramaswamy2001multiclass}, for example, tens of thousands of genes expression data are measured per patient, far exceeding the number of patients. Such practical settings motivate the study of high-dimensional statistics, where certain structures of the data are exploited to make statistical inference possible. One representative example is sparse linear models~\cite{hastie2015statistical}, where we assume that a linear regression task's underlying predictor depends only on a small subset of the input features.

%for example, in personalized medicine, high dimensional features of patients can

On the other hand, online learning with bandit feedback, due to its practicality in many applications such as online news recommendations~\cite{li10acontextual} or clinical trials~\cite{liao2016sample, woodroofe1979one}, has attracted a surge of research interests.
Of particular interest is linear bandits, where in $n$ rounds, the learner repeatedly takes an action $A_t$ (e.g., some feature representation of a product or a medicine) from a set of available actions $\cA \subset \RR^d$ and receives a reward $r_t = {\langle\theta^*,A_t\rangle} + \eta_t$ as feedback where  $\eta_t \in \RR$ is an independent zero-mean, $\sigma$-sub-Gaussian noise.
Sparsity structure is abundant in linear bandit applications: for example, customers' interests on a product depend only on a number of its key specs; the effectiveness of a medicine only depends on a number of key medicinal properties, which means that the unknown parameter $\theta^*$ sparse; i.e., it has a small number of nonzero entries.

%$$r_t = \inner{\theta^*}{A_t} + \eta_t, $$
%\kj{maybe ``''?}
%\chicheng{I rewrote the paragraph above. Kyoungseok and Kwang, please check if you have a chance!} \kj{done!}
%\chicheng{Set $\sigma = 1$ here, to be consistent with the preliminaries section?}
%\chicheng{Small comment: it would be good if we are consistent with using $a^\T b$ or $\inner{a}{b}$.}

%In many applications of linear bandits,
%\cz{I recommend compressing the first two paragraphs into just one, and start introducing examples that are related to experimental design / bandits. How about something like this: We can use e.g. \url{https://hastie.su.domains/StatLearnSparsity_files/SLS_corrected_1.4.16.pdf} to find more examples
%} \ja{I also think your introduction is much better than my one. I replaced the starting of intro to yours}

%The bandit field was no exception, and the problem called sparse linear bandit is now one of the mainstream in the linear bandit field. Let $\mathcal{A} \subset \mathbb{R}^{d}$ be the action set. For each round $t$, the agent chooses an actions $a_t \in \mathcal{A}$ and then receives a reward $r_t$ as a noisy linear function:
%    $$r_t = a_t^\top \theta^* + \eta_t $$
%where $\eta_t \in \mathbf{R}$ is a $\sigma$ sub-Gaussian noise. The objective of the agent is to maximize its cumulative rewards.

%Starting from
% people notice that
%bandit algorithms can achieve some informational advantage from sparsity.

%\cz{Here, we can also discuss the difference of our setup with other lines of works, e.g. the ones with stochastic contexts~\cite{oh2021sparsity,kim19doubly,bastani2020online,sivakumar2020structured,liu2020smoothed,pmlr-v80-wang18j,wang2020nearly} - basically those that appear in Hao et al's Table 1 and follow up works.
%}
%\ja{I used it on the related works section now...... However, currently I am considering about the structure of the introduction...... should I have to include related works in introduction part?} \cz{I see - I agree that these may be more suitable for ``related work'' section.}

%\begin{itemize}
%  \item $\sqrt{sdn} $ is optimal, but it requires $n \ge ds$, and there is no known computationally efficient procedure for the generic arm set.
%  \item however, we know that a good set of measurements can accelerate the recovery. for example, L1 estimation error or the minimum signal.
%  \item thus the idea of having a nonvacuous regret bound for $n = o(d)$  and a computationally efficient procedure is quite attractive. here, the regret bound must naturally be arm set dependent .
%  \item hao et al. did this but there were a few open problems. first, the gap between the upper and lower bounds. second, the arm set dependent quantity.
%\end{itemize}

Early studies~\cite{ay12online,carpentier2012bandit,lattimore2015linear} on sparse linear bandits have revealed that leveraging sparsity assumptions yields bandit algorithms with lower regret than those provided by full-dimensional linear bandit algorithms~\cite{abe99associative,auer02using,dani08stochastic,abbasi2011improved}.
However, most existing studies either rely on a particular arm set (e.g., a norm ball), which is unrealistic in many applications, or use computationally intractable algorithms.
If we consider an arbitrary arm set, however, the optimal worst-case regret is ${\Theta}(\sqrt{sdn})$ where $s$ is the sparsity level of $\th^*$, which means that as long as $n = O(sd)$, there exists an instance for which the algorithm suffers a linear regret~\cite{lattimore18bandit}.
This is in stark contrast to supervised learning where it is possible to enjoy nontrivial prediction error bounds for $n = o(d)$~\cite{foster94risk}.
This motivates a natural research question: Can we develop computationally efficient {sparse linear bandit} algorithms that allow a generic arm set  yet enjoy nonvacuous bounds in the data-poor regime by exploiting problem-dependent characteristics?

The seminal work of~\citet{hao2020high} provides a positive answer to this question.
They propose algorithms that enjoy nonvacuous regret bounds with an arbitrary arm set in the data poor regime using Lasso.
Specifically, they have obtained a regret bound of $\tilde O({\Cmin}^{-2/3} s^{2/3}n^{2/3})$ where ${\Cmin}$ is an arm-set-dependent quantity.
However, their work still left a few open problems.
First, their regret upper bound does not match with their lower bound $\Omega({\Cmin}^{-1/3} s^{1/3}n^{2/3})$.
Second, it is not clear if ${\Cmin}$ is the right problem-dependent constant that captures the geometry of the arm set.

%In this work,
%
%
%To get around this issue and still enjoy a meaningful regret bound for the data poor regime of $n=o(d)$, one may go beyond the minimax regret and attempt to derive a regret bound that depends on the geometry of the given arm set or the known parameter $\th^*$.
%
%
%existing high-dimensional regression algorithms like best subset selection (BSS) or Lasso can enjoy
%

%\chicheng{I made a pass on this part - some (usused) texts are commented out here.}
%Though it is a special case, Recalling that the limit in the linear bandit, which is the parent topic of the sparse linear bandit, is $\Theta(d\sqrt{n})$\cite{dani08stochastic, abbasi2011improved, lattimore18bandit}, we can say that those results are ideal result as much as the algorithm already knows where the nonzero coordinate is.
%design one of the earliest analysis about the sparse linear bandit studies which
%the matching upper bound to the known regret lower bound of the
%for the exploration
%\red{
%Early studies of~\citet{ay12online,carpentier2012bandit,lattimore2015linear} reveal that leveraging sparsity assumptions yields bandit algorithms with lower regret than those provided by full-dimensional linear bandit algorithms~\cite{dani08stochastic, abbasi2011improved, lattimore18bandit}.
%\citet{ay12online} devised the \textrm{SeqSEW} algorithm which has a near-optimal $\Omega(\sqrt{sdn})$ regret bound for sparse linear bandit. However, their algorithm was computationally intractable. \cite{carpentier2012bandit, lattimore2015linear} studied the special cases where the action set is the unit $\ell_2$ ball or a scaled hypercube, respectively and proved $\tilde{O}(s\sqrt{n})$ regret bounds.
%}

%interesting
%studies is
%sparse linear bandit
%In high-dimensional regression problems, usually the feature dimension $d$ is a big quantity that can be even larger than the number of samples.
%Therefore it could be beneficial to remove the $d$ term from the regret order analysis.
%number of interaction rounds
%\red{
%A recent work~\citep{hao2020high} initiated the research on designing
%sparse linear bandit algorithms with general (fixed) action sets that have dimension-free regret guarantees. This is motivated by high-dimensional linear bandit applications where the feature dimension $d$ can even be larger than $n$, the time horizon length, in which case even a $O(\sqrt{sdn})$ regret bound is unaffordable.
%\cite{hao2020high} proposed an explore-then-commit-based algorithm, ESTC, for this setting. They show a dimension-free regret upper bound that depends on  $\mathcal{C}_{\min}$, a constant that captures the geometry of the action set. In addition, they proved the first $\Omega(n^{2/3})$ dimension-free regret lower bound for sparse linear bandits.
%relieved the dimension dependency of the regret upper bound by introducing
%for the fixed action set setting
% constant instead
% geometry

%points for improvement
%The pioneering study~\cite{hao2020high} left out several open questions. First, its regret upper and lower bounds are not matching, and it was left as an open problem that what is the minimax optimal regret bound order. Moreover, there was a gap between theoretical estimation error and practical exploration planning for the bandit algorithm. The algorithm ESTC is mainly based on Lasso \cite{tibshirani96regression}, which has a $\ell_1$ norm bound error based on the compatibility constant \cite{wainwright2019high,bv11}. However, it is difficult to compute the compatibility constant with current techniques, and they used $\mathcal{C}_{\min}$ instead, which is easier to compute but provides a looser analysis. See Table \ref{table: results} for a recap of the results in \citet{hao2020high}.}

%In particular, as it was known that Lasso~\cite{tibshirani96regression}, a celebrated $\ell_1$-regularization method, has the effect of naturally and efficiently controlling the number of non-zero coordinates in sparse linear models, many research methods are based on Lasso. Plus, due to the mathematical characteristics of the Lasso, people generally assume eigenvalue-related conditions such as restricted eigenvalue condition or compatibility condition for better results.

%However, by focusing only on the decomposition of the dot product through Holder's inequality and the $\ell_1$ norm guarantee of Lasso, many other papers failed to consider the predictive error bound for a fixed point. In particular, although Lasso has its own disadvantages such as bias, previous sparse linear bandit researches did not consider much about other estimators other than Lasso.
%\cz{I think the above discussion on lasso is better put to the ``comparison with lasso'' section.}

\begin{table}[t]
%--------- "\ding{55} \ding{51}"
% \vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
% \setlength\tabcolsep{4.1pt}
\begin{tabular}{lcccr}
\toprule
 & Regret Bound & Data-poor & Assumptions \\
\midrule
\citet{hao2020high} & $\tilde{O}(s^{2/3}\mathcal{C}_{\min}^{-2/3}n^{2/3})$ & \ding{51} & $\mathcal{A}$ spans $\mathbb{R}^d$ \\
\citet{hao2020high} & ${\Omega(s^{1/3}\kappa^{-2/3}n^{2/3})}$ &\ding{51} & $\cA$ spans $\RR^d$ \\
% (\autoref{sec:etc-wpopart})
Algorithm~\ref{alg:etc-sparse} (Ours)  & $\tilde{O}(s^{2/3} H_*^{2/3}n^{2/3})$ & \ding{51} & $\mathcal{A}$ spans $\mathbb{R}^d$\\
% (\autoref{sec:phased-elim-wpopart})
Theorem~\ref{thm:lower} (Ours) & $\Omega(s^{2/3}\kappa^{-2/3}n^{2/3})$ &\ding{51} & $\cA$ spans $\RR^d$\\
\hline
\citet{hao2020high} & $\tilde{O}(\sqrt{\mathcal{C}_{\min}^{-1}sn})$ & \ding{55} & $\mathcal{A}$ spans $\mathbb{R}^d$, Min. Signal \\
Algorithm~\ref{alg:phase-elim} (Ours) & $\tilde{O}(\sqrt{sn})$ & \ding{55} & $\mathcal{A}$ spans $\mathbb{R}^d$, Min. Signal\\
%\xmark \cmark
\bottomrule
\end{tabular}
\end{sc}
\caption{
  Regret bounds of our work and the prior art where $s$, $d$, $n$ are the sparsity level, the feature dimension, and the number of rounds, respectively.
  The quantities $\mathcal{C}_{\min}$ and $H_*^2$ are the constants that captures the geometry of the action set (see Eq.~\eqref{def: Cmin} and~\eqref{def: H2}), and $\kappa$ is a parameter for a specific family of arm sets that satisfies $ \kappa^{-2} =  \Theta( \mathcal{C}_{\min}^{-1}) = \Theta( H^2_* )$.
%   \kj{do we want to say $\kappa^2 \le H^2_*$ as well?}
  In general, $ H_*^2 \leq \mathcal{C}_{\min}^{-1} \le \mathcal{C}_{\min}^{-2}$ (Propositon~\ref{prop:H2 vs Cmin}).
%\kj{TODO: use $\kappa$ but explain that $\kappa=\mathcal{C}_{\min}=H^2$}
%\kj{$T$ to $n$}
% \kj{lower order terms are omitted for brevity.}
}
\label{table: results}
\end{small}
\end{center}
% \vskip -0.1in
\end{table}

%\chicheng{Ideally (if we have time) we should also mention some historical contexts of sparse linear regression and experimental design, and how we improve over those. But it is perhaps secondary.}
In this paper, we make significant progress in high-dimensional linear regression and sparse linear bandits, which resolves or partly answers the aforementioned open problems.

\textbf{First} (Section~\ref{sec:popart}), we propose a novel and computationally efficient estimator called \ensuremath{\textsc{PopArt}}(POPulation covariance regression with hARd Thresholding) that enjoys a tighter $\ell_1$ norm recovery bound than the de facto standard sparse linear regression method Lasso in many problems.
Motivated by the $\ell_1$ norm recovery bound of \ensuremath{\textsc{PopArt}}, we develop a computationally-tractable design of experiment objective for finding the sampling distribution that minimize the error bound of \ensuremath{\textsc{PopArt}}, which is useful in settings where we have control  on the sampling distribution (such as compressed sensing).
Our design of experiments results in an $\ell_1$ norm error bound that depends on the measurement set dependent quantity denoted by $H_*^{2}$ (see Eq.~\eqref{def: H2} for precise definition) that is provably better than ${\Cmin}^{-1}$ that appears in the $\ell_1$ norm error bound used in \citet{hao2020high}, thus leading to an improved planning method for {sparse linear} bandits.
\textbf{Second} (Section~\ref{sec:bandits}), Using \ensuremath{\textsc{PopArt}}, we design new algorithms for the sparse linear bandit problem, and improve the regret upper bound of prior work~\cite{hao2020high}; see Table~\ref{table: results} for the summary.
\textbf{Third} (Section~\ref{sec:lower-bound}), We prove a matching lower bound in data-poor regime, showing that the regret rate obtained by our algorithm is optimal.
The key insight in our lower bound is a novel application of the algorithmic symmetrization technique \cite{simchowitz2017simulator}. {Unlike the conjecture of \citet[Remark 4.5]{hao2020high}, the improvable part was not the algorithm but the lower bound for sparsity $s$.}

We empirically verify our theoretical findings in Section~\ref{sec:expr} where \ensuremath{\textsc{PopArt}} shows a favorable performance over Lasso.
Finally, we conclude our paper with future research enabled by \ensuremath{\textsc{PopArt}} in Section~\ref{sec:conclusion}.
For space constraint, we discuss related work in Appendix~\ref{sec:related} but closely related studies are discussed in depth throughout the paper.

%\red{Finally, we verify our theoretical improvement of the algorithm by the experiment.  } \kj{in terms of L1 recovery guarantee of \popart and the sparse linear bandit problems.}

%\kj{====}

%
%Specifically,
%\begin{itemize}
%    \item We propose a novel and computationally efficient estimator called \popart (POPulation covariance regression with hARd Thresholding) that enjoys a tighter $\ell_1$ norm recovery bound than the de facto standard sparse linear regression method Lasso in many problems. % compared to prior approaches like Lasso.
%    \item Motivated by the $\ell_1$ norm recovery bound of \popart, we develop a computationally-tractable design of experiment objective for finding the sampling distribution that minimize the error bound of \popart, which can is useful in settings where we have control  on the sampling distribution (such as compressed sensing).
%    %and
%    %make the following
%    % which achieves a performance improvement by calculating variance for each coordinate and thresholding, instead of the traditional lasso-based estimation.
%    \item Using \popart, we design new algorithms for the sparse linear bandit problem, and improve the regret upper bound of prior work~\cite{hao2020high}.
%    \item We prove a matching lower bound in data-poor regime, showing that the regret rate obtained by our algorithm is optimal.
%    The key insight in our lower bound is a novel application of the algorithmic symmetrization technique \cite{simchowitz2017simulator}. \red{Unlike the speculation of \citet[Remark 4.5]{hao2020high}, the improvable part was not the algorithm but the lower bound. }\kj{see if hao et al. is saying UB can be improved; if true, then say `unlike their speculation'}
%    \item \red{Finally, we verify our theoretical improvement of the algorithm by the experiment.  } \kj{in terms of L1 recovery guarantee of \popart and the sparse linear bandit problems.}
%    % which is about the optimal order of the instance dependent constant.
%\end{itemize}
%We summarize our theoretical results and their comparison with~\citet{hao2020high} in Table~\ref{table: results}.

\vspace{-7pt}
\section{Problem Definition and Preliminaries} \label{sec: prelim}
\vspace{-6pt}

\noindent\textbf{Sparse linear bandits.} We study the sparse linear bandit learning setting, where the learner is given access to an action space ${\cA} \subset \{a\in\RR^d: \|a\|_\infty  \le 1\}$, and repeatedly interacts with the environment as follows: at each round $t = 1,\ldots,n$, the learner chooses some action $A_t \in \cA$, and receives reward feedback $r_t = {\langle\theta^*,A_t\rangle} + \eta_t$, where ${\theta^*} \in \RR^d$ is the underlying reward predictor, and ${\eta_t}$ is an independent zero-mean $\sigma$-subgaussian noise.
We assume that $\theta^*$ is $s$-sparse; that is, it has at most $s$ nonzero entries.
The goal of the learner is to minimize its pseudo-regret defined as
\[
\Reg(n) = n \max_{a \in \cA} {\langle\theta^*,a\rangle} - \sum_{t=1}^n {\langle\theta^*,A_t\rangle}.
\]
%maximize its cumulative reward \sum_{t=1}^T
%\subsection{Experimental design}
%(called experiments)
% \cz{Adding this for more background.}
\noindent\textbf{Experimental design for linear regression.} In the experimental design for linear regression problem, one has a pool of unlabeled examples $\cX$, and some underlying predictor $\theta^*$ to be learned.
%Each example $x$ is associated with a label .
Querying the label of $x$, i.e. conducting experiment $x$, reveals a random label $y = {\langle\theta^*,x\rangle} + \eta$ associated with it, where $\eta$ is a zero mean noise random variable. The goal is to accurately estimate $\theta^*$, while using as few queries $x$ as possible.

%nice
% with

%In the following description, the experimental design plays an important role.

% The following definition describes some of the key designs and constants.

\begin{definition}(Population covariance matrix $Q$)
Let $\mathcal{P}({\cX})$ be the space of probability measures over $\mathcal{X}$ with the Borel $\sigma$-algebra, and define the population covariance matrix for the distribution $\mu \in \mathcal{P}(\cX)$ as follows:
\begin{equation}
Q(\mu):=\int_{a \in \mathcal{X}} a a^\top d \mu(a)
\end{equation}
Classical approaches for experimental design focus on finding a distribution $\mu$ such that its induced population covariance matrix $Q(\mu)$ has properties amenable for building a low-error estimator, such as D-, A-, G-optimality~\cite{fedorov2013theory}.

\textbf{Compatibility condition for Lasso.~}
%
For a positive definite matrix $\Sigma \in \RR^{d\times d}$ and a sparsity level $s\in[d] := \{1,\ldots,d\}$, % set of indices $S \subset [d]$,
we define its compatibility constant $\phi_0^2 (\Sigma, s)$
% \cz{Should this be the compatibility constant of a particular subset $S$ or all subset of size $s$? Also, shall we emphasize this dependence using notation $\phi_0^2(\Sigma, s)$, similar to~\cite{bickel2009simultaneous}? }\ja{done. Sorry for the confusion}
as follows:
\begin{equation}
\phi_0^2 (\Sigma, s):=\min_{S\subseteq[d]: |S| = s}~ \min_{v: \| v_{S} \|_1 \leq 3\| v_{-S} \|_1}\frac{s v^\top \Sigma v}{\| v_S \|_1^2},\label{def: comp const}
\end{equation}
where $v_S \in\RR^d$ denotes the vector that agrees with $v$ in coordinates in $S$ and $0$ everywhere else and $v_{-S}\in\RR^d$ denotes $v - v_S$.

\textbf{Notation.~}
Let $e_i$ be the $i$-th {canonical basis} vector.
We define $[x] = \{1,2,\ldots,x\}$.
Let $\supp(\theta)$ be the set of coordinate indices $i$ where $\theta_i \neq 0$.
We use $a \lesssim b $ to denote that there exists an absolute constant $c$ such that $a \le cb$.

%\cz{As discussed in the meeting, perhaps use
%\[
%\phi_0^2
%:=
%\min_{S: |S| = s} \min_{v: \| v_{-S} \|_1 \leq 3\| v_{-S} \|_1}
%\frac{s v^\top \Sigma v}{\| v_S \|_1^2},
%\]
%for more explicit expression.
%}\ja{Change applied. Thanks for the fix.}
\end{definition}

\section{Improved Linear Regression and Experimental Design for Sparse Models}
\label{sec:popart}
% algorithm
%\cz{Proposed change of title: } \ja{Change applied. Thanks for your idea. }
%namely
%namely

In this section, we discuss our novel sparse linear estimator \ensuremath{\textsc{PopArt}} for the setting where the population covariance matrix is known and show its strong theoretical properties.
We then present a variation of \ensuremath{\textsc{PopArt}} called \ensuremath{\textsc{Warm-PopArt}} that amends a potential weakness of \ensuremath{\textsc{PopArt}}, followed by our novel experimental design for \ensuremath{\textsc{PopArt}} and discuss its merit over prior art.

%\ja{ Thanks for your writing about the section introduction. I write it on here officially.}

%\cz{Now that I think about organizing this section, does it make more sense to use the following order: (1) \popart (with general $\mu$); (2) \wpopart (still with general $\mu$; (3) new experimental design criterion? This way, we make \wpopart and experimental design decoupled. And in the subsequent bandit algorithm, we can let the algorithm for solve the optimal experimental design problem to get $\mu^*$ and run $\wpopart$ on $\mu^*$.} \ja{Done. Changed the order of the section and few words}

%\popart
\textbf{\ensuremath{\textsc{PopArt}}(POPulation covariance regression with hARd Thresholding).}
% \kj{remove the subsection; but word smithing around the beginning of the next paragraph.}
Unlike typical estimators for the statistical learning setup, our main estimator \ensuremath{\textsc{PopArt}} described in Algorithm~\ref{alg:popart} takes the population covariance matrix denoted by $Q$ as input.
We summarize our assumption for \ensuremath{\textsc{PopArt}}.
% Specifically, in \popart, we assume that the given data points ${(\cova_t,\resp_t)}_{t=1}^n$ satisfy that $\cova_t \sr{\text{i.i.d.}}{\sim} \mu$ from some distribution $\mu$ and that $Q = Q(\mu) := \EE_{X\sim \mu}[X X^\T]$.
% Furthermore, we assume that $\resp_t = \inner{\theta^*}{\cova_t} + \eta_t$ with $\eta_t$ being zero-mean $\sig$-subgaussian noise.
% % where $\cova_t$'s are drawn iid from $\mu$ and $\resp_t = \inner{\theta^*}{\cova_t} + \eta_t$ with $\eta_t$ being a zero-mean 1-subgaussian noise;

\begin{assumption}\label{ass:popart}
(Assumptions on the input of \ensuremath{\textsc{PopArt}})
There exists $\mu$ such that the input data points $\{(X_t,Y_t)\}_{t=1}^n$ satisfy that $X_t \sr{\text{i.i.d.}}{\sim} \mu$ and $Q=Q(\mu) := \EE_{X\sim \mu}[X X^\T]$.
Furthermore, $Y_t = {\langle\theta^*,X_t\rangle} + \eta_t$ with $\eta_t$ being zero-mean $\sig$-subgaussian noise.
Also, $R_0 \geq \max_{a \in \mathcal{A}} |\langle a, \theta^* - \theta_0 \rangle|$.
\end{assumption}

% \chicheng{I see the phrase ``pilot estimator'' used in \url{https://arxiv.org/abs/1604.08098v3} - so it is not completely ungrounded.}

%\chicheng{For experimental design for linear regression, shall we use $(A_t, r_t)$ notation or shall we use $(x_t, y_t)$ notation?}

%\chicheng{Rewrote the algorithm's different stages and some motivations on algorithm design - please check.}
\begin{algorithm}[h]
\caption{ \ensuremath{\textsc{PopArt}}(POPulation covariance regression with hARd Thresholding)}
\label{alg:popart}
\begin{algorithmic}[1]
% A_2, \cdots, A_t
%number of samples $n$,
% \STATE \textbf{Input:} Samples $(\cova_t, \resp_t)_{t=1}^n$, where $(\cova_t)_{t=1}^n$ are drawn iid from $\mu$, population covariance matrix $Q(\mu) = \EE_{\cova \sim \mu} [\cova \cova^\top]$, pilot estimator $\theta_0 \in \mathbb{R}^d$, failure rate $\delta$, $R_0$, an upper bound of $\max_{a \in \mathcal{A}} |\langle a, \theta^* - \theta_0 \rangle|$
\STATE \textbf{Input:} Samples $\{(X_t, Y_t)\}_{t=1}^n$, the population covariance matrix $Q\in\RR^{d\times d}$, pilot estimator $\theta_0 \in \mathbb{R}^d$, an upper bound $R_0$ of $\max_{a \in \mathcal{A}} |\langle a, \theta^* - \theta_0 \rangle|$, failure rate $\delta$.
\STATE \textbf{Output:} estimator $\hat{\theta}$
\FOR{$t=1,\ldots,n$ }
\STATE $\tilde{\theta}_t = Q^{-1}X_t (Y_t-\langle X_t, \theta_0\rangle )+ \theta_0$
\label{step:one-sample-estimator}
\ENDFOR
% \FOR{$i=1,\ldots, d$}
\STATE $\forall i\in[d], {\theta}_i'=\textsf{Catoni}(\{\tilde{\theta}_{ti}:=\langle\tilde{\theta}_{t}, e_i \rangle\}_{t=1}^n , \alpha_i, \frac{\delta}{2d})$  where $\alpha_i:= \sqrt{\frac{2\log \frac{2d}{\delta}}{n(R_0^2 + \sigma^2)(Q^{-1})_{ii}(1+ \frac{2\log \frac{2d}{\delta}}{n-2 \log \frac{2d}{\delta}})}}$
\label{step:catoni-i}
% \ENDFOR
\STATE $\hat{\theta}\leftarrow \textsf{clip}_\lambda ({\theta'}):= [{\theta}'_i \one(|{\theta}'_i|>\lambda_i)]_{i=1}^d$ where $\lambda_i$ is defined in Proposition~\ref{prop:individual conf bound}.
\label{step:hard-threshold}
\RETURN $\hat{\theta}$
\end{algorithmic}
\end{algorithm}

\ensuremath{\textsc{PopArt}} consists of several stages. In the first stage, for each $(X_t, Y_t)$ pair, we create a one-sample estimator $\tilde{\theta}_t$ (step~\ref{step:one-sample-estimator}).
The estimator, $\tilde{\theta}_t$, can be viewed as a generalization of doubly-robust estimator~\cite{chernozhukov2019semi,dudik2011doubly} for linear models. %$\theta^*$.
Specifically, it is the sum of two parts: one is the pilot estimator $\theta_0$ that is a hyperparameter of \ensuremath{\textsc{PopArt}}; the other is $Q(\mu)^{-1} X_t (Y_t-\langle X_t, \theta_0 \rangle )$, an unbiased estimator of the difference $\theta^* - \theta_0$.
Thus, it is not hard to see that $\tilde{\theta}_t$ is an unbiased estimator of $\theta^*$.
As we will see in Theorem~\ref{thm: main bounds of estimator}, the variance of $\tilde{\theta}_t$ is smaller when $\theta_0$ is closer to $\theta^*$, showing the advantage of allowing a pilot estimator $\theta_0$ as input.
If no good pilot estimator is available a priori, one can set $\theta_0=0$.

%This stage
%instead of the already
From the discussion above, it is natural to take an average of $\tilde{\theta}_t$. Indeed, when $n$ is large, the population covariance matrix $Q(\pi)$ is close to empirical covariance matrix $\hat{Q} := \frac1n \sum_{t=1}^n X_t X_t^\top$, which makes $\hat{\theta}_{\text{avg}} := \frac1n \sum_{t=1}^n \tilde{\theta}_t$ close to the ordinary least squares estimator $\hat{\theta}_{\text{OLS}} = \hat{Q}^{-1} (\frac1n\sum_{t=1}^n X_t Y_t)$.
However, for technical reasons, the concentration property of $\tilde{\theta}_{\text{avg}}$ is hard to establish.
This motivates \ensuremath{\textsc{PopArt}}'s second stage (step~\ref{step:catoni-i}), where, for each coordinate $i \in [d]$, we employ
Catoni's estimator \cite{lugosi2019mean} (see Appendix~\ref{sec:catoni} for a recap) to obtain an intermediate estimate for each $\theta_i^*$, namely $\theta_i'$.

%H^2 (Q)

%;  for details
To use Catoni's estimator, we need to have an upper bound of the variance of $\theta_i'$ for its $\alpha_i$ parameter. A direct calculation yields that, for all $i\in[d]$ and $t\in[n]$,  $$\textrm{Var}(\tilde{\theta}_{ti})\leq \del{ \max_{a \in \cA} {\langle\theta^*-\theta_0,a\rangle}^2  + \sigma^2 } \max_i (Q(\mu)^{-1})_{ii}$$ where $\tilde{\theta}_{ti}:=\langle \tilde{\theta}_t,e_i\rangle$. This implies that $\del{ R_0^2  + \sigma^2 } \max_i (Q(\mu)^{-1})_{ii}$ is an upper bound of $\textrm{Var}(\tilde{\theta}_{ti})$. By the standard concentration inequality of Catoni's estimator (see Lemma~\ref{lem:catoni-error}), we obtain the following estimation error guarantee for $\theta_i'$; the proof can be found in Appendix~\ref{appendix:proof-prop}.
Hereafter, all proofs are deferred to appendix unless noted otherwise.

%(see Proposition~\ref{prop:individual conf bound})
%which in turn  on $\max_{a \in \cA} \inner{\theta^*-\theta_0}{a}$; we choose to use \chicheng{todo for CZ: explain $H^2(Q)$.}

%The parameter $\alpha$ in Catoni's estimator, relies on

% Definition \ref{def: Catoni}
% for the definition
%\kj{write some connecting sentence, introducing this result.}
% at least
\begin{proposition} \label{prop:individual conf bound}
Suppose Assumption~\ref{ass:popart} holds.
In \ensuremath{\textsc{PopArt}}, for $i\in [d]$, if $n\geq 2 \ln \frac{2d}{\delta}$, the following inequality holds with probability $1-\frac{\delta}{d}$:
$$|\theta_i' - \theta_i^*|< \sqrt{\frac{4(R_0^2 + \sigma^2) (Q(\mu)^{-1})_{ii}^2}{n} \log \frac{2d}{\delta}} =: \lambda_i$$
\end{proposition}
%\begin{proof}
%See .
%\end{proof}
%The proof proceeds by combining concentration bound of the Catoni's esimator with the computation of the variance of our estimator, we get the following proposition, which shows the main advantage of this approach. The full proof can be found in

%\kj{thanks to Proposition 1, it is natural to perform hard thresholding ; mention that ``easy to see that with high probability we do not have false positives''}

%The second stage ensures that $\theta'$ and $\theta^*$ are close in $\ell_\infty$ norm.
%To exploit the sparsity of $\theta^*$,

%By the coordinate-wise error guarantee of $\theta'$,
%\kj{how about just ``confidence interval''?}
%notation?}
Proposition~\ref{prop:individual conf bound} shows that,
for each coordinate $i$,
$(\theta_i' - \lambda_i, \theta_i' + \lambda_i)$ forms a confidence interval for $\theta_i^*$.
Therefore, if $0 \notin (\theta_i' - \lambda_i, \theta_i' + \lambda_i)$, we can infer that $\theta_i^* \neq 0$, i.e., $i \in \supp(\theta^*)$.
Based on the observation above, \ensuremath{\textsc{PopArt}}'s last stage (step~\ref{step:hard-threshold})
performs a hard-thresholding for each of the coordinates of $\theta'$, using the threshold $\lambda_i$ for coordinate $i$.
Thanks to the thresholding step, with high probability, $\hat{\theta}$'s support is contained in that of $\theta^*$, which means that all coordinates $i$ outside the support of $\th^*$ (typically the vast majority of the coordinates when $s \ll d$) satisfy $\hat\theta_i = \theta^*_i = 0$.
Meanwhile, for coordinate $i$'s in $\supp(\theta^*)$, the estimated value $\hat{\theta}_i$ is not too far from $\theta^*_i$.

%We remark that replacing hard thresholding with soft thresholding also enjoys a similar $\ell_1$-closeness guarantee.

%%an additionally achieves an $\ell_1$-closeness to $\theta^*$.
%$\hat{\theta}$ preserves the $\ell_\infty$ closeness to $\theta^*$,
%$\theta_i'$, with threshold  to obtain its final estimator, $\hat{\theta}$.
% with essentially the same guarantee
%\chicheng{Minor comment: soft thresholding also works here with essentially the same guarantee.}
% and its corresponding reward $r_1, r_2, \cdots, r_t$
% perspective
The following theorem states \ensuremath{\textsc{PopArt}}'s estimation error bound in terms of its output  $\hat{\theta}$'s $\ell_\infty$, $\ell_0$, and $\ell_1$ errors, respectively. We remark that replacing hard thresholding in the last stage with soft thresholding enjoys similar guarantees.
%
%Let $H^2(Q):=\max_{i \in [d]} (Q^{-1})_{ii}$, , $\beta := \sqrt{\frac{4 (R_{0}^2 + \sigma^2)H^2(Q)}{n}\log \frac{2d}{\delta}}$, and $\alpha:= \sqrt{\frac{2\log \frac{1}{\delta}}{n(R_0^2 + \sigma^2)H^2(Q)(1+ \frac{2\log \frac{1}{\delta}}{n-2 \log \frac{1}{\delta}})}} $.
%If the threshold $\lambda > \beta$ and $n \geq 2\ln\frac{2d}{\delta}$,
\begin{theorem}\label{thm: main bounds of estimator}
Take Assumption~\ref{ass:popart}.
Let $H^2(Q):=\max_{i \in [d]} (Q^{-1})_{ii}$.
%If \popart receives inputs $\cbr{\cova_t, \resp_t}_{t=1}^n$ drawn from $\mu$, $Q(\mu)$, \gray{pilot estimator $\theta_0$, failure rate $\delta$,}\kj{seems unnecessary?} and $R_0$ such that $R_0 \geq \max_{a \in \mathcal{A}} |\langle a, \theta^* - \theta_0 \rangle|$, then all the following items hold
Then, \ensuremath{\textsc{PopArt}} has the following guarantees
with probability at least $1-\delta$: %\kj{similar to my comment in proposition 1, we can just define the assumption with theorem environment and then refer to it}
%\chicheng{As discussed in today's meeting, let's assume $n \geq 2\ln\frac{2d}{\delta}$, and use the observation that $n - 2\ln\frac{2d}{\delta} \geq \frac n 2$ to simplify the above theorem's presentation.} \ja{Done.}
\begin{enumerate}[label=(\roman*)]
    \item $\forall i \in [d], |\hat{\theta}_i - \theta_i^*|< 2 \sqrt{\frac{4 (R_0^2 + \sigma^2)(Q(\mu)^{-1})_{ii}}{n}\log \frac{2d}{\delta}}$ so $\|\hat{\theta}-\theta^*\|_\infty < 2 \sqrt{\frac{4 (R_0^2 + \sigma^2)H^2(Q(\mu))}{n}\log \frac{2d}{\delta}}, $
    \item $\textrm{supp}(\hat{\theta})\subset \textrm{supp}({\theta}^*)$ so $\|\hat{\theta}-\theta^*\|_0 \leq s$,
    \item $\|\hat{\theta}-\theta^*\|_1 \leq 2s \sqrt{\frac{4 (R_0^2 + \sigma^2)H^2(Q(\mu))}{n}\log \frac{2d}{\delta}}$
    %\chicheng{$\lambda := \sqrt{\frac{4 (\bar{R}^2 + \sigma^2)H^2(Q)}{n}\log \frac{2d}{\delta}}$}
\end{enumerate}
\end{theorem}
Interestingly, \ensuremath{\textsc{PopArt}} has no false positive for identifying the sparsity pattern and  enjoys an $\ell_\infty$ error bound, which is not available from Lasso, to our knowledge.
Unfortunately, a direct comparison with Lasso is nontrivial since the largest compatibility constant $\phi_0^2 (\hat{\Sigma}, s)$ is defined as the solution of the optimization problem~\eqref{def: comp const}, let alone the fact that $\phi_0^2 (\hat{\Sigma}, s)$ is a function of the empirical covariance matrix.
While we leave further investigation as future work, our experiment results in Section~\ref{sec:expr} suggest that there might be a case where \ensuremath{\textsc{PopArt}} makes a meaningful improvement over Lasso.
\vspace{-0.25cm}
\begin{proof}[Proof of Theorem~\ref{thm: main bounds of estimator}]
{Let $\lambda := \max_i \lambda_i = \sqrt{\frac{4(R_0^2 + \sigma^2) H^2(Q(\mu))}{n} \log \frac{2d}{\delta}}$}
From Proposition \ref{prop:individual conf bound} and the union bound, one can check that \begin{equation} \label{eqn: result of prop1}
    \|{\theta'}-\theta^*\|_\infty < \lambda
\end{equation} with probability $1-\delta$. Therefore, the coordinates in $\textrm{supp}(\theta^*)^c$ will be thresholded out because of $\|{\theta'}-\theta^*\|_\infty \leq \lambda$. Therefore, (ii) holds and for all $i \in \textrm{supp}(\theta^*)^c$, $|\hat{\theta}_i - \theta_i^*|=0$.

By definition, $\hat{\theta}=\textsf{clip}_\lambda ({\theta'})$, we can say that $\|\hat{\theta}-\theta'\|_\infty \leq \lambda$. Plus, by Eq. (\ref{eqn: result of prop1}), $\|\theta'-\theta^* \|_\infty \leq \lambda$. By the triangle inequality, $\|\theta^*-\hat{\theta}\|_\infty \leq 2 \lambda$. Therefore, (i) holds.

%Now for the coordinates in $\textrm{supp}(\hat{\theta})$, they are not thresholded out so for $i \in \textrm{supp}(\hat{\theta})$, $\hat{\theta}_i = \tilde{\theta}_i$. Therefore, $|\hat{\theta}-\theta_i^*|\leq \beta$. It remains to check the coordinates in $\textrm{supp}(\hat{\theta})^c \cap \textrm{supp}(\theta^*)$. Since $\|\tilde{\theta}-\theta^* \|_\infty < \lambda$, for $j \in \textrm{supp}(\hat{\theta})^c \cap \textrm{supp}(\theta^*)$, $|\hat{\theta}_j - \theta_j^* | \leq \|\hat{\theta}_j - \tilde{\theta}_j\|_\infty + \|\tilde{\theta}_j - \theta_j^* \| \leq \lambda+ \beta$. Therefore, (i) holds.
% summing up the above result makes
Lastly, (iii) can be argued as follows:
%\begin{align*}
\[
    \|\hat{\theta}-\theta^*\|_1 = \sum_{i \in [d]} |\hat{\theta}_i - \theta_i^*|
    \leq \sum_{i \in \textrm{supp}(\theta^*)^c} 0 + \sum_{i \in \textrm{supp}(\theta^*)} 2\lambda \leq 2 s \lambda.
    %\qquad
    \qedhere
\]
%\end{align*}
\end{proof}
\vspace{-8pt}

% \subsection{\wpopart: improving the guarantees of \popart by a two-stage procedure}
% \label{sec:wpopart}
% % estimator

\textbf{\ensuremath{\textsc{Warm-PopArt}}: Improved guarantee by warmup.~}
%
One drawback of the \ensuremath{\textsc{PopArt}} estimator is that its estimation error scales with $\sqrt{R_0^2 + \sigma^2 }$, which can be very large when $R_0$ is large.
One may attempt to use the fact that \ensuremath{\textsc{PopArt}} allows a pilot estimator $\theta_0$ to address this issue since $R_0$ gets smaller as $\th_0$ is closer to $\th^*$.
However, it is a priori unclear how to obtain a $\theta_0$ close to $\theta^*$ as $\theta^*$ is the unknown parameter that we wanted to estimate in the first place. %needs to be estimated.

%, is the algorithm that
To get around this ``chicken and egg'' problem, we propose to introduce a warmup stage, which we call \ensuremath{\textsc{Warm-PopArt}}(Algorithm \ref{alg:warm-popart}).
\ensuremath{\textsc{Warm-PopArt}} consists of two stages.
For the first warmup stage, the algorithm runs $\ensuremath{\textsc{PopArt}}$ with the zero vector as the pilot estimator and with the first half of the samples to obtain a coarse estimator denoted by $\hat\theta_0$  which guarantees that for large enough $n_0$, $\|\hat\theta_0 - \theta^*\|_1 \leq \sigma$. In the second stage, using $\hat\theta_0$ as the pilot estimator, it runs $\ensuremath{\textsc{PopArt}}$ on the remaining half of the samples.

% initial point
%We summarize the details in Algorithm \ref{alg:warm-popart}.
%algorithm
%, not $\sigma$ as other confidence bound theorems do. Usually this is a minor issue compare to the order of $n, d$, or $s$, but one can avoid this problem if $R_0 \approx \sigma$, or $\|\theta^* - \theta_0\|_1 \leq \sigma$.
%\chicheng{}

%This can be done by the proper choice of the $\theta_0$ with spending only few samples on it.

%Input: Samples $\cova_1, \cova_2, \cdots, \cova_{n_0} \sim \mu$ and its corresponding reward $\resp_1, \resp_2, \cdots, \resp_{n_0}$,
\begin{algorithm}[h]
\caption{\ensuremath{\textsc{Warm-PopArt}}}
\begin{algorithmic}[1]
% \STATE \gray{\textbf{Input:}
% Samples $(\cova_t, \resp_t)_{t=1}^{n_0}$, where $(\cova_t)_{t=1}^n$ are drawn iid from $\mu$,
% population covariance matrix $Q(\mu) = \EE_{A \sim \mu} [A A^\top]$ (abbrev. $Q$),
% number of samples $n_0$, failure rate $\delta$, $R_{\max}$, an upper bound of $\max_{a \in \cA} |\langle \theta^* , a \rangle| $.}

\STATE \textbf{Input:} Samples $\{(X_t, Y_t)\}_{t=1}^{n_0}$, the population covariance matrix $Q\in\RR^{d\times d}$, an upper bound
$R_{\max}$ of $\max_{a \in \cA} |\langle \theta^* , a \rangle| $, number of samples $n_0$, failure rate $\delta$.

%which is
%reward upper bound
%\{r_i\}_{i=1}^{n_0},
%,\{r_i\}_{i=\lfloor \frac{n_0}{2} \rfloor +1}^{n_0}
\STATE \textbf{Output:} $\hat{\theta}$, an estimate of $\theta^*$
\STATE Run $\ensuremath{\textsc{PopArt}}( \{( X_i, Y_i )\}_{i=1}^{\lfloor n_0/2 \rfloor}, Q,  \undefined {0}, \delta, R_{\max})$ to obtain ${\hat\theta_0}$, a coarse estimate of $\theta^*$ for the next step.
\label{step:coarse-estimation}

\STATE Run $\ensuremath{\textsc{PopArt}}( \{( X_i, Y_i )\}_{i= \lfloor n_0/2 \rfloor +1}^{n_0},Q, \hat\theta_0, \delta, \sigma)$ to obtain $\hat{\theta}$, an estimate of $\theta^*$.
\end{algorithmic}
\label{alg:warm-popart}
\end{algorithm}

The following corollary states the estimation error bound of the output estimator $\hat{\theta}$. Compared with \ensuremath{\textsc{PopArt}}'s $\ell_1$ recovery guarantee, \ensuremath{\textsc{Warm-PopArt}}'s $\ell_1$ recovery guarantee (Equation~\eqref{eqn:warm-popart-l1}) has no dependence on $R_{\max}$; its dependence on $R_{\max}$ only appears in the lower bound requirement for $n_0$.

%Note that there's no $R_{\max}$ term in the regret bound now.

%\chicheng{Note: the current version still requires the algorithm to have the knowledge of $R_{\max}$. To get rid of this requirement, one standard idea is to set $\lambda_0$ to something like $\sigma^2 (1/n_0)^{\frac13}$ (the exponent needs to be strictly smaller than $\frac12$); so long as $\lambda_0 \geq 2\sqrt{\frac{2(R_{\max}^2 + \sigma^2) H^2(Q)}{n_0}\log \frac{2d}{\delta}}$, and $s \lambda_0 \leq \sigma^2$ (which can be simultaneously satisfied when $n_0$ is larger than some constant threshold), we can still guarantee that the first stage gives $\hat{\theta}$ such that $\| \hat{\theta} - \theta^* \|_1 \leq \sigma$, ensuring the success of the second stage. But perhaps this is a bit too much - we should just assume $R_{\max}$ is known? Or add an remark such that we know how to get rid of rmax.
%}

\begin{corollary}\label{cor:warm-popart}
Take Assumption~\ref{ass:popart} without the condition on $R_0$.
% Suppose that \wpopart receives inputs $\cbr{\cova_t, \resp_t}_{t=1}^{n_0}$ drawn from $\mu$, $Q(\mu)$, failure rate $\delta$, and $R_{\max}$ such that
Assume that $R_{\max} \geq \max_{a \in \mathcal{A}} |\langle a, \theta^* \rangle|$, and $n_0>\frac{32s^2(R_{\max}^2 + \sigma^2)H^2 (Q(\mu))}{\sigma^2} \log \frac{2d}{\delta}$.
Then, \ensuremath{\textsc{Warm-PopArt}} has, with probability at least $1-2\delta$,
%all the following items hold
%Let $H^2(Q):=\max_{i \in [d]} (Q^{-1})_{ii}$, $\lambda_0 := 2\sqrt{\frac{2(R_{\max}^2 + \sigma^2) H^2(Q)}{n_0}\log \frac{2d}{\delta}}$ and $\lambda_1 := 4\sigma\sqrt{\frac{H^2(Q)}{n_0}\log \frac{2d}{\delta}}$. If $n_0>\max(2\frac{4(R^2 + \sigma^2)H_*^2 (Q)}{\sigma^2} \log \frac{2d}{\delta}, 4\log \frac{2d}{\delta}))$, \kj{$R$ to $R_{\max}$.} then all the following bound holds with probability at least $1-2\delta$:
\begin{equation}
    \|\hat{\theta}-\theta^*\|_1 \leq 8 s \sigma  \sqrt{\frac{H^2(Q(\mu))\ln \frac{2d}{\delta}}{n_0}}.
    \label{eqn:warm-popart-l1}
\end{equation}
\end{corollary}
%\kj{should we use $H^2(Q(\pi))$?}
%. For example, it could be the traditional
% based exploration - in this case the first exploration phase length changes to $\frac{8 s^2 \sigma^2 \log \frac{2d}{\delta}}{\lambda_{\min} (Q)^2}$ where
\begin{remarks}
In Algorithm~\ref{alg:warm-popart}, we choose $\ensuremath{\textsc{PopArt}}$ as our coarse estimator, but we can freely change the coarse estimation step (step~\ref{step:coarse-estimation}) to other principled estimation methods (such as Lasso) without affecting the main estimation error bound~\eqref{eqn:warm-popart-l1}; the only change will be the lower bound requirement of $n_0$ to another problem-dependent constant.
\end{remarks}
%On the other hand, one could also just remove the first warm-up phase ($n_0=0$) and let $\theta_0=\vec{0}$. This saves the exploration time especially when $R_{\max} = {O}(\sigma)$.

%all the following items hold
%knowledge of
\begin{remarks} \ensuremath{\textsc{Warm-PopArt}} requires the knowledge of $R_{\max}$, an upper bound of $\max_{a \in \cA} |\langle \theta^* , a \rangle|$; this requirement can be relaxed by changing the last argument of the coarse estimation step (step~\ref{step:coarse-estimation}) from $R_{\max}$, to some function $f(n_0)$ such that $f(n_0) = \omega(1)$ and $f(n_0) = o(\sqrt{n_0})$ (say, $\sigma n_0^{\frac14}$);
with this change, a result analogous to Corollary~\ref{cor:warm-popart} can be proved with a different lower bound requirement of $n_0$.
\end{remarks}
%\chicheng{Make a pass on remark 2}

%,
%for deciding the threshold $\lambda_0$, and even though this is quite a standard in linear bandit studies,
% \subsection{A new optimal experimental design criterion for sparse linear estimation}
% \label{sec:opt-expt-design}
\textbf{A novel and efficient experimental design for sparse linear estimation.~}
%
%We can enjoy these results more
%when we can control the distribution of the sample $\mu$, and naturally,
%the population covariance matrix $Q(\mu)$. We can think about the experimental design for minimizing $H^2 (Q)$, or formally,
%Now, define the quantities of the action set geometry as follows:
In the experimental design setting where the learner has freedom to design the underlying sampling distribution $\mu$, the $\ell_1$ error bound of \ensuremath{\textsc{PopArt}} and \ensuremath{\textsc{Warm-PopArt}} naturally motivates a design criterion.
Specifically, we can choose $\mu$ that minimizes $H^2(Q(\mu))$, which gives the lowest estimation error guarantee.
We denote the optimal value of $H^2(Q(\mu))$ by
\begin{gather}
    H_*^2 := \underset{\mu \in \mathcal{P}(\cA)}{\textrm{min}}\max_{i\in[d]} (Q(\mu)^{-1})_{ii} ~. \label{def: H2}
\end{gather}

The minimization of $H^2(Q(\mu))$ is a convex optimization problem, which admits efficient methods for finding the solution.
Intuitively, $H_*^2$ captures the geometry of the action set $\cA$.%: when $\cA$ approximately lies in an low-dimensional subspace, $H_*^2$ will be large, as for all $\mu$, $Q(\mu)^{-1}$ will be large entrywise.  \kj{I feel it is rather misleading our selling point. We expect to have a small value of $H_*^2$ for the rademacher arm set, but that one does not lies in a low dimensional subspace!}
%so we can approximate efficiently

%This approach provides not only a control over $\ell_\infty$ norm and support, but also a tighter $\ell_1$ bound in terms of the action set geometry.
%perform when composed with

%\gray{How does the $\ell_1$ recovery guarantee of \popart with this experimental design, compare with those of prior works, specifically Lasso?}

To compare with previous studies that design a sampling distribution for Lasso, we first review the standard $\ell_1$ error bound of Lasso.
\begin{theorem}
\label{thm:lasso}
(\citet[Theorem 6.1]{bv11}) With probability at least $1-2\delta$, the $\ell_1$-estimation error of the optimal Lasso solution $\hat{\theta}_{\Lasso}$ \cite[Eq. (2.2)]{bv11} with $\lambda = \sqrt{2\log(2d/\delta)/n}$ satisfies
$$ \| \hat{\theta}_{\Lasso}-\theta^*\|_1 \leq \frac{s \sigma}{\phi_0^2 (\hat{\Sigma}, s)}\sqrt{\frac{2 \log (2d/\delta) }{n}},$$

where $\phi_0 (\hat{\Sigma}, s)^2$ is the compatibility constant with respect to the empirical covariance matrix $\hat{\Sigma} = \frac{1}{n} \sum_{t=1}^n X_t X_t^\top$  and the sparsity $s$ in Eq. \eqref{def: comp const}.
\end{theorem}

Ideally, for Lasso, experiment design which minimizes the compatibility constant will guarantee the best estimation error bound within a fixed number of samples $n$. However, naively, the computation of the compatibility constant is intractable since Eq.~\eqref{def: comp const} is a combinatorial optimization problem which is usually difficult to compute.
One simple approach taken by~\citet{hao2020high} is to use the following computationally tractable surrogate of $\phi_0^2 (\hat{\Sigma}, s)$:
\begin{gather}
    \mathcal{C}_{\min} := \underset{\mu \in \mathcal{P}(\cA)}{\textrm{max}}\lambda_{\tmin} (Q(\mu)) \label{def: Cmin}
\end{gather}
where $\lambda_{\tmin}(A)$ denotes the minimum eigenvalue of a matrix $A$.
With the choice of sampling distribution $\mu = \underset{\mu \in \mathcal{P}(\cA)}{\textrm{argmax}}\lambda_{\tmin} (Q(\mu))$, and $n \geq \tilde{\Omega}(\frac{s \cdot \mathrm{polylog}(d)}{{\Cmin}^2})$, with high probability, $\phi_0^2 (\hat{\Sigma},s) \geq \Cmin/2$ holds~\cite[][Theorem 1.8]{rudelson2012reconstruction},
%\cite{javanmard2014confidence}
and one can replace $\phi_0 (\hat{\Sigma}, s)$ to ${\Cmin}/2$ in Theorem \ref{thm:lasso} to get the following corollary:
\begin{corollary}
\label{cor:lasso with Cmin} With probability at least $1-\exp(-c n)-2\delta$ for some universal constant $c$, the $\ell_1$-estimation error of the optimal Lasso solution $\hat{\theta}_{\Lasso}$ satisfies
\begin{equation}\label{eqn:lasso with Cmin}
    \| \hat{\theta}_{\Lasso}-\theta^*\|_1 \leq \frac{2s \sigma}{{\Cmin}}\sqrt{\frac{2 \log (2d/\delta) }{n}},
\end{equation}
\end{corollary}

%%\chicheng{Original result is from Rudelson and Zhou? I'll check.}

%$\mathcal{C}_{\min}$ has several desirable properties - With high probability the inequality $\frac{\mathcal{C}_{\min}}{2}<\phi_0^2$ holds, and $\mathcal{C}_{\min}$ is concave so it is easy to approximate.
%$\mathcal{C}_{\min}$ as a
%, defined as follows

%In terms of this perspective,
%Compared with the guarantees obtained by experimental design for Lasso, our new estimator has a better estimation error bound compare to the previous approaches, as shown the proposition below whose proof can be found in Appendix \ref{appendix: example of H2 and Cmin}. \kj{to me, it is easier to read if we say ``
The following proposition shows that our estimator has a better error bound compared to the surrogate experimental design for Lasso of~\citet{hao2020high}.

%
\begin{proposition}\label{prop:H2 vs Cmin}
We have $ H_*^2 \leq \mathcal{C}_{\min}^{-1} \leq d H_*^2$. Furthermore, there exist arm sets for which either of the inequalities is tight up to a constant factor.
\end{proposition}

Therefore, our new estimator has $\ell_1$ error guarantees at least a factor $\mathcal{C}_{\min}^{-1/2}$ better
than that provided by~\cite{hao2020high}, as follows: when we choose the $\mu$ as the solution of the Eq. \eqref{def: H2}, then
$$ (\text{RHS of \eqref{eqn:warm-popart-l1}}) \lesssim s \sigma H_* \sqrt{\frac{\ln(2d/\delta)}{n}} \lesssim s \sigma {\Cmin}^{-1/2} \sqrt{\frac{\ln(2d/\delta)}{n}} \lesssim s\sigma {\Cmin}^{-1} \sqrt{\frac{\ln(2d/\delta)}{n}} \lesssim (\text{RHS of \eqref{eqn:lasso with Cmin}})$$

In addition, we also prove that there exists a case where our estimator has an $d/s$-order better error bound compared to the traditional lasso bound in Theorem~\ref{thm:lasso}, although this is not in terms of the compatibility constant of the empirical covariance matrix $\hat{\Sigma}$.

\begin{proposition}\label{prop:H2 vs compat} There exists an action set $\mathcal{A}$ and an absolute constant $C_1>0$ such that $$H_* <C_1 \frac{s}{d} \times \frac{1}{\phi_0^2 (\Sigma, s)}$$
\end{proposition}
For the detailed proof about Proposition \ref{prop:H2 vs Cmin} and Proposition \ref{prop:H2 vs compat}, see Section \ref{appendix: example of H2 and Cmin} in Appendix.

%instance dependent term at least for the order of .

\section{Improved Sparse Linear Bandits using \ensuremath{\textsc{Warm-PopArt}}}
\label{sec:bandits}

%The bandit algorithm is the best field that can utilize this estimator.
%In bandit literature, one of the major approaches for the algorithm is Explore-then-Commit (ETC) algorithm, especially designing the first exploration phase based on the user's taste. On the other hand, the user has to shorten the unnecessary exploration as much as possible to ensure sufficient time for exploitation.
% by getting an accurate estimation within a shorter time

We now apply our new \ensuremath{\textsc{Warm-PopArt}} sparse estimation algorithm to design new sparse linear bandit algorithms.
Following prior work~\cite{hao2020high}, we adopt the  classical Explore-then-Commit (ETC) framework for algorithm design, and use \ensuremath{\textsc{PopArt}} with experimental design to perform exploration.
As we will see, the tighter $\ell_1$ estimation error bound of our \ensuremath{\textsc{PopArt}}-based estimators helps us obtain an improved regret bound.

\begin{algorithm}[h]
\caption{Explore then commit with \ensuremath{\textsc{Warm-PopArt}}}
\begin{algorithmic}[1]
\STATE Input: time horizon $n$, action set $\cA$, warm-up exploration length $n_0$, failure rate $\delta$, reward threshold parameter $R_{\max}$, an upper bound of $\max_{a \in \cA} |\langle \theta^* , a \rangle| $.
%
\STATE Solve the optimization problem in Eq.~\eqref{def: H2} and denote the solution as $\mu_*$
\FOR{$t=1, \ldots, n_0$}
\STATE Independently pull the arm $A_t$ according to $\mu_*$ and receives the reward $r_t$
\ENDFOR
\STATE Run $\ensuremath{\textsc{Warm-PopArt}}(\{A_t\}_{t=1}^{n_0}, \{r_t\}_{t=1}^{n_0}, Q(\mu_*), \delta, R_{\max})$ to obtain $\hat{\theta}$, an estimate of $\theta^*$.
\FOR{$t=n_0 +1,\ldots,n$}
\STATE Take action $A_t = \argmax_{a \in \cA} {\langle\hat{\theta},a\rangle}$, receive reward $r_t = {\langle\theta^*,A_t\rangle} + \eta_t$
\ENDFOR
\end{algorithmic}
\label{alg:etc-sparse}
\end{algorithm}

\textbf{Sparse linear bandit with \ensuremath{\textsc{Warm-PopArt}}.~}
% \label{sec:etc-wpopart}
%
% its actions from $\mu_*$,
Our first new algorithm, Explore then Commit with \ensuremath{\textsc{Warm-PopArt}}(Algorithm~\ref{alg:etc-sparse}), proceeds as follows. For the exploration stage, which consists of the first $n_0$ rounds, it solves the optimization problem~\eqref{def: H2} to find $\mu_*$, the optimal sampling distribution for $\ensuremath{\textsc{PopArt}}$ and samples from it to collect a dataset for the estimation of $\theta^*$. Then, we use this dataset to compute the \ensuremath{\textsc{Warm-PopArt}} estimator $\hat{\theta}$. Finally, in the commit stage, which consists of the remaining $n-n_0$ rounds, we take the greedy action with respect to $\hat{\theta}$. We prove the following regret guarantee of Algorithm~\ref{alg:etc-sparse}:

%for the rest of the rounds.

\begin{theorem}
\label{thm:etc-sparse}
If Algorithm~\ref{alg:etc-sparse} has input time horizon $n>16\sqrt{2}\frac{R_{\max} (R_{\max}^2 + \sigma^2)^{3/2} H_*^2 s^2}{\sigma^4} \log \frac{2d}{\delta}$, action set $\cA \subset [-1,+1]^d$, and exploration length $n_0 = 4(s^2 \sigma^2 H_*^2 n^2 \log \frac{2d}{\delta}R_{\max}^{-2})^{\frac{1}{3}}$, $\lambda_1 = 4\sigma \sqrt{\frac{H_*^2}{n_0}\log \frac{2d}{\delta}}$, then with probability at least $1-2\delta$,
$
\Reg(n) \leq 8R_{\max}^{1/3}(s^2 \sigma^2 H_*^2 n^2 \log \frac{2d}{\delta})^{\frac{1}{3}}
$.
\end{theorem}
%the regret is bounded as follows

\begin{proof}
From Corollary \ref{cor:warm-popart}, $ \|\hat{\theta} - \theta^*\|_1 \leq 2s \lambda_1$ with probability at least $1-2\delta$. Therefore, with probability $1-2\delta$,
\begin{align*}
    \textrm{Reg}(n) &\leq  R_{\max} n_0 + (n-n_0)\|\hat{\theta} - \theta^*\|_1 \leq R_{\max} n_0 + 2sn\lambda_1 = R_{\max} n_0 + 8sn\sigma \sqrt{\frac{H_*^2}{n_0}\log \frac{2d}{\delta}}
\end{align*}
and optimizing the right hand side with respect to $n_0$ leads to the desired upper bound.
\end{proof}

%Overall, the regret upper bound is of the order ,

Compared with~\citet{hao2020high}'s regret bound $\tilde{O}((R_{\max} s^2 \sigma^2 {\Cmin}^{-2} n^2)^{1/3})$\footnote{This is implicit in~\cite{hao2020high} -- they assume that $\sigma=1$ and do not keep track of the dependence on $\sigma$.}
, Algorithm~\ref{alg:etc-sparse}'s regret bound $\tilde{O}((R_{\max} s^2 \sigma^2 H_*^2 n^2)^{1/3})$
is at most $\tilde{O}((R_{\max} s^2 \sigma^2 {\Cmin}^{-1} n^2)^{1/3})$, which is at least a factor ${\Cmin}^{\frac13}$ smaller. As we will see in Section~\ref{sec:lower-bound}, we show that the regret upper bound provided by Theorem~\ref{thm:etc-sparse} is unimprovable in general, answering an open question of~\cite{hao2020high}.

%is a factor of $(H_* C_{\min})^{\frac23}$ smaller, which is $\leq (\sqrt{C_{\min}})^{\frac23} \leq 1$.

%Recall that $1 \leq H_* \leq \frac{1}{\sqrt{C_{\min}}}$

%$\mathcal{C}_{\min}^{-2/3}$ to $H_*^{2/3}$, at least of the order $\mathcal{C}_{\min}^{-1/3}$. This gap of the instance-dependent constant order was one of the open problems for this sparse linear bandit in data-poor regime, and we optimized the regret upper bound.
%previous research for the sparse linear bandit problem
% \label{sec:phased-elim-wpopart}
%
\textbf{Improved upper bound with minimum signal condition.~} Our second new algorithm, Algorithm~\ref{alg:phase-elim}, similarly uses \ensuremath{\textsc{Warm-PopArt}} under an additional minimum signal condition.

\begin{assumption}[Minimum signal]There exists a known lower bound $m > 0$ such
that $\min_{j\in \textrm{supp}(\theta^*)} |\theta_j^*| > m$.
\end{assumption}

At a high level, Algorithm~\ref{alg:phase-elim} uses the first $n_2$ rounds for identifying the support of $\theta^*$; the $\ell_\infty$ recovery guarantee of $\ensuremath{\textsc{Warm-PopArt}}$ makes it suitable for this task. Under the minimal signal condition and a large enough $n_2$, it is guaranteed that $\hat{\theta}_2$'s support equals exactly the support of $\theta^*$. After identifying the support of $\theta^*$, Algorithm~\ref{alg:phase-elim} treats this as a $s$-dimensional linear bandit problem by discarding the remaining $d-s$ coordinates of the arm covariates,
and perform phase elimination algorithm \citep[Section 22.1]{lattimore18bandit} therein. The following theorem provides a regret upper bound of Algorithm \ref{alg:phase-elim}.

\begin{algorithm}[h]
\caption{Restricted phase elimination with \ensuremath{\textsc{Warm-PopArt}}}
\begin{algorithmic}[1]
\STATE Input: time horizon $n$, finite action set $\cA$, minimum signal $m$, failure rate $\delta$, reward threshold parameter $R_{\max}$, an upper bound of $\max_{a \in \cA} |\langle \theta^* , a \rangle|$
\STATE Solve the optimization problem in Eq. \ref{def: H2} and denote the solutions as $Q$ and $\mu_*$, respectively.
\STATE Let $n_2 = \max(\frac{256\sigma^2 H_*^2}{m^2} \log \frac{2d}{\delta} , \frac{32s^2(R_{\max}^2 + \sigma^2)H_*^2}{\sigma^2} \log \frac{2d}{\delta})$
\FOR{$t=1,\ldots,n_2$}
    \STATE Independently pull the arm $A_t$ according to $\mu_*$ and receives the reward $r_t$
\ENDFOR
\STATE $\hat{\theta}_2 = \ensuremath{\textsc{Warm-PopArt}}(\{ A_t\}_{t=1}^n, \{ R_t\}_{t=1}^n, Q, \delta,R_{\max})$
\STATE Identify the support $\hS = \mathrm{supp}(\hat{\theta}_2)$
\FOR{$t=n_2+1,\ldots,n$}
\STATE Invoke phased elimination algorithm for linear bandits on $\hS$
\ENDFOR
\end{algorithmic}
\label{alg:phase-elim}
\end{algorithm}

\begin{theorem} \label{thm:with minimum signal}
If Algorithm \ref{alg:phase-elim} has input time horizon $n>\max(\frac{2^8\sigma^2 H_*^2}{m^2} , \frac{2^5 s^2(R_{\max}^2 + \sigma^2)H_*^2}{\sigma^2} )\log \frac{2d}{\delta}$, action set $\cA \subset [-1,1]^d$, upper bound of the reward $R_{\max}$, then with probability at least $1-2\delta$, the following regret upper bound of the Algorithm \ref{alg:phase-elim} holds: for universal constant $C>0$,
$$ \textrm{Reg} (n) \leq \max(\frac{2^8\sigma^2 H_*^2}{m^2} \log \frac{2d}{\delta} , \frac{2^5s^2 (R_{\max}^2 + \sigma^2)H_*^2}{\sigma^2} \log \frac{2d}{\delta}) + C\sigma\sqrt{sn \log (|\cA|n)}$$

\end{theorem}
%\chicheng{why do we need this?}

%In the special case that $m >16\sigma H_* (sn)^{-1/4} \log^{1/2} \frac{2d}{\delta}$
For sufficiently large $n$, the second term dominates, and we obtain an $O(\sqrt{sn})$ regret upper bound. Theorem~\ref{thm:with minimum signal} provides two major improvements compared to~\citet[][Algorithm 2]{hao2020high}. First,
when $m$ is moderately small (so that the first subterm in the first term dominates),
it shortens the length of the exploration phase $n_2$ by a factor of $s \cdot \frac{{\Cmin}}{H_*^2}$. Second, compared with the regret bound
$\tilde{O}( \sqrt{\frac{9\lambda_{\max}(\sum_{i=1}^{n_2}A_i A_i^\top/n_2)}{\mathcal{C}_{\min}}} \sqrt{sn} )$ provided by~\cite{hao2020high},
our main regret term $\tilde{O}(\sqrt{sn})$ is more interpretable and can be much lower.

%$, and , a factor of $
%improves over the regret bound of
%~\cite{hao2020high}, which is random, highly probable to be greater than $1$, and could be extraordinarily problematic.
%We could achieve these improvements thanks to the superiority of $\textrm{PopArt}$. $\textrm{PopArt}$ assures the confidence bound of each coordinate, so there's no need to add redundant analysis for calculating the confidence about the support of $\hat{\theta}$.
% action set dependent constant $\frac{9\lambda_{max}(\sum_{i=1}^{n_2}A_i A_i^\top/n_2)}{\mathcal{C}_{\min}}$ from

\vspace{-6pt}
\section{Matching lower bound}
\label{sec:lower-bound}
\vspace{-6pt}
We show the following theorem that establishes the optimality of Algorithm~\ref{alg:etc-sparse}. This solves the open problem of \citet[][Remark 4.5]{hao2020high} on the optimal order of regret in terms of sparsity and action set geometry in sparse linear bandits.
\begin{theorem}\label{thm:lower}
For any algorithm, any $s, d, \kappa$ that satisfies
$d \geq \max (n^{1/3} s^{4/3} \kappa^{-4/3},(s+1)^2)$ and $n>8\kappa s^2$, there exists a linear bandit environment an action set $\cA$ and a $s$-sparse $\theta \in \RR^d$, such that $\mathcal{C}_{\min}^{-1} \leq \kappa^{-2}$, $R_{\max} \leq 2$, $\sigma = 1$, and
%\max\del{ {H_*^2(\cA)}, \mathcal{C}_{\min}(\cA)^{-1} }
\[
\textrm{Reg}_n \geq \Omega( \kappa^{-2/3} s^{2/3} n^{2/3})~.
\]
\end{theorem}
%\chicheng{Additionally discuss that this implies Theorem~\ref{thm:etc-sparse} has a matching lower bound.}

%\begin{theorem}
%\chicheng{Add the requirement for $n, d, s, H_*$}
%for any policy $\pi$,
%\chicheng{(optional) for any $R_{\max}$ - it is also OK to stick to current version with $R_{\max} = O(1)$}
%there exists an action set $\mathcal{A}$ with $H_*^2 > 0$ and $s$-sparse parameter $\th \in \mathbb{R}^d$ such that $$ \textrm{Reg}_n \geq \Omega( \red{R_{\max}^{1/3}} H_*^{2/3} s^{2/3} n^{2/3})$$
%\end{theorem}
We give an overview of our lower bound proof techniques, and defer the details to Appendix \ref{sec:proof-lb}.
% and technical lemmas are in
%Appendix: lower bound

%One of the recent technique for proving the lower bound is
\noindent\textbf{Change of measure technique.~}
Generally, researchers prove the lower bound by comparing two instances based on the information theory inequalities, such as  Pinsker's inequality, or Bregtanolle-Huber inequality. In this proof, we also use two instances $\theta$ and $\theta'$, but we use the change of measure technique, to help lower bound the probability of events more freely. Specifically, for any event $A$,
\begin{align}\label{eqn:change-of-measure}
    \PP_\th (A)
    = \EE_{\th} [\one_A]
    =
    \EE_{\th'}\sbr{ \one_A \prod_{t=1}^n\frac{p_\th(r_t |a_t)}{p_{\theta'}(r_t |a_t)} }
    \gtrsim \EE_{\theta'} \sbr{ \one_A \exp\del{-\sum_{t=1}^n \langle A_t , \theta-\theta' \rangle^2} }~.
\end{align}
%= \EE_{\th} [\one_A]
%This approach help us  more freely.

\noindent\textbf{Symmetrization.~}
We utilize the algorithmic  symmetrization technique of~\citet{simchowitz2017simulator, bubeck11pure-tcs}, which makes it suffice to focus on proving lower bounds against  symmetric algorithms.
%, and extending the result to the general algorithms.

\begin{definition}[Symmetric Algorithm]
An algorithm $\textsf{Alg}$ is \emph{symmetric} if for any permutation $\pi \in \textit{Sym}(d)$, $\theta \in \mathbb{R}^{d}$, $\{a_t\}_{t=1}^n \in \mathcal{A}^n$,
$$ \PP_{\theta, \textsf{Alg}} (A_1 = a_1, \cdots , A_n = a_n) = \PP_{{\pi} (\theta) , \textsf{Alg}}(A_1 = \pi(a_1 ), \cdots , A_n = \pi(a_n ))$$
where for vector $v$, $\pi(v) \in \RR^d$ denotes its permuted version that moves $v_i$ to the $\pi(i)$-th position.
%\chicheng{$$ \EE_{\theta, \textsf{Alg}} \sbr{f(A_1, \ldots, A_n)} = \EE_{{\pi} (\theta) , \textsf{Alg}} \sbr{ f( \pi^{-1}(A_1), \ldots, \pi^{-1}(A_n) ) }$$}
\end{definition}
%advantage of
This approach can help us to exploit the symmetry of $\theta'$ to lower bound the right hand side of~\eqref{eqn:change-of-measure}; below, $\Pi := \cbr{\pi': \pi(\theta') = \theta'}$ is the set of permutations that keep $\theta'$ invariant, {and $A$ is an event invariant under $\Pi$}:
\begin{align*}
    \text{~\eqref{eqn:change-of-measure}}
    \geq
    \frac{1}{|\Pi|} \sum_{\pi \in \Pi}
    \EE_{ \theta'} \sbr{ \one_A \exp(-\sum_{t=1}^n \langle \pi^{-1}(A_t) , \theta-\theta' \rangle^2) }    \geq
    \EE_{ \theta'} \sbr{ \one_A \exp\del{ -\sum_{t=1}^n \frac{1}{|\Pi|} \sum_{\pi \in \Pi} \langle \pi^{-1}(A_t) , \theta-\theta' \rangle^2} }
\end{align*}
which helps us use combinatorial tools over the actions for the lower bound proof.

% \paragraph{Averaging hammer over action instead of $\theta$} Symmetrization provides us the possibility to deal with the actions, but it also has its own drawback that it cannot control $\theta$ much. Previous approaches usually utilized the `averaging hammer', or `minimum is smaller than the average' trick. It was usually done as follows: when we want to calculate the upper bound of $\EE_\theta [f(\bA)]$ (usually it is about the informational divergence) and want the case when this bound is small,

% \begin{itemize}
%     \item Choose $\theta$ that minimize $\EE_\theta [f(\bA)]$ among the set of possible candidates $\Theta$.
%     \item Calculate $\frac{1}{|\Theta|} \sum_{\theta' \in \Theta}\EE_{\theta'} [f(\bA)]$. It is better to set $\Theta$ to have symmetry to make calculation easier.
%     \item Since $\theta$ is the minimizing argument, $\EE_\theta [f(\bA)] \leq \frac{1}{|\Theta|} \sum_{\theta' \in \Theta}\EE_{\theta'} [f(\bA)]$.
% \end{itemize}
% However, this approach chooses specific hidden instance that we never know explicitly, which harms the symmetry. Instead, we find a way to use this 'averaging hammer' over the actions. Keen readers would notice that, for the symmetric algorithm $\textsf{Alg}$ and a permutation which satisfies $\pi(\theta)=\theta$, $\PP_{\theta, \textsf{Alg}} (\bA) = \PP_{\theta, \textsf{Alg}} (\pi (\bA))$. Therefore, we did 'averaging hammer' in this way.

% \begin{itemize}
%     \item For each $\bA \in \mathcal{A}$, make a partition (more precisely, an equivalence class) $\Gamma_i$ such that $\PP_{\theta, \textsf{Alg}} (\bA) = \PP_{\theta, \textsf{Alg}} (\bA')$. Because of the symmetric algorithm assumption, we can find a lot of them.
%     \item To calculate $\EE_\theta [f(\bA)]$ with a convex function $f$, the following holds:
%     \begin{align*}
%         \EE_\theta[ f(\bA)] &= \sum_{\bA \in \cA^n} \PP_\theta(\bA) f(\bA)= \sum_{\Gamma_i} \sum_{\bA\in \Gamma_i} \PP_\theta(\bA) f(\bA)\leq \sum_{\Gamma_i} \sum_{\bA\in \Gamma_i} \PP_\theta(\bA) f(\frac{1}{|\Gamma_i|}\sum_{\bA\in \Gamma_i}\bA)
%     \end{align*}
%     where the last inequality is from Jensen for each $\Gamma_i$.
%     \item With the symmetry, it is easier to calculate the average over each partition $\Gamma_i$
% \end{itemize}

% It provides much tighter calculation about the lower bound for the symmetric algorithms and maximize the advantage of symmetrization.

\vspace{-6pt}
\section{Experimental results}
\label{sec:expr}
\vspace{-6pt}

We evaluate the empirical performance of \ensuremath{\textsc{PopArt}} and our proposed experimental design, along with its impact on sparse linear bandits. One can check our code from here: \url{https://github.com/jajajang/sparse}.

\begin{figure}[h]
    \centering
    \begin{tabular}{cc}
           \toprule
            Case 1 & Case 2 \\
            \midrule
                 \begin{tabular}{l}
                 \includegraphics[width=0.4\linewidth]{figures/l1reg_threefigures_largerfont.png}
                 \end{tabular}&
                 \begin{tabular}{l}
                 \includegraphics[width=0.4\linewidth]{figures/d30_l1reg_largerfont_three.png}
                 \end{tabular}
    \\%\midrule
    \begin{tabular}{l}
    \includegraphics[width=0.4\linewidth]{figures/bandit_t400000_largerfont.png}\end{tabular}&\begin{tabular}{l} \includegraphics[width=0.4\linewidth]{figures/d30_bandit_largerfont.png}
    \end{tabular}
   \\\bottomrule
        \end{tabular}
    \caption{Experiment results on $\ell_1$ estimation error cumulative regret. %We run \popart with the solution of Eq. \ref{def: H2}.
      %Cmin-Lasso is the algorithm of \cite{hao2020high} that solves ... Eq. xx, and H2-Lasso is the algorithm that performs exploration based on the solution of Eq. \ref{def: H2}.
    %   \kj{increase the resolution; you can do it by exporting the plot to pdf from python and then use pdf viewer to enlarge it and then screencapture it}
    }
    \label{fig:table_of_figures}
\end{figure}

For sparse linear regression and experimental design, we compare our algorithm \ensuremath{\textsc{PopArt}} with $\mu$ being the solution of~\eqref{def: H2} with two baselines.
The first baseline denoted by $C_{\min}$-Lasso is the method proposed by~\citet{hao2020high} that uses Lasso with sampling distribution $\mu$ defined by~\eqref{def: Cmin}.
The second baseline is $H^2$-Lasso, uses Lasso with sampling distribution $\mu$ defined by~\eqref{def: H2}, which is meant to observe if Lasso can perform better with our experimental design and to see how \ensuremath{\textsc{PopArt}} is compared with Lasso as an estimator since they are given the same data.
Of course, this experimental design is favored towards \ensuremath{\textsc{PopArt}} as we have optimized the design for it, so our intention is to observe if there ever exists a case where \ensuremath{\textsc{PopArt}} works better than Lasso.

% how Lasso and \popart performs when the how the performance of Lasso is compared with

% estimation

%Algorithm~\ref{alg:etc-sparse}
For sparse linear bandits, we run a variant of our Algorithm~\ref{alg:etc-sparse} that uses \ensuremath{\textsc{Warm-PopArt}} in place of \ensuremath{\textsc{PopArt}} for simplicity.
As a baseline, we use ESTC~\cite{hao2020high}.
For both methods, we use the exploration length prescribed by theory.
We consider two cases:
\begin{itemize}
    \item \textbf{Case 1: Hard instance where $H_*^2 \ll \mathcal{C}_{\min}^{-1}$.~} We use the action set constructed in Appendix \ref{example:worst case of Cmin and H2} where $H_*^2$ and $\mathcal{C}_{\min}$ shows a gap of $\Theta(d)$. We choose $d=10$, $s=2$, $\sigma=0.1$.
    \item \textbf{Case 2. General unit vectors.~} In this case, we choose $d=30$, $s=2$, $\sigma=0.1$ and the action set $\mathcal{A}$ consists of $|\mathcal{A}|=3d=90$ uniformly random vectors on the unit sphere.
\end{itemize}

We run each method 30 times and report the average and standard deviation of the $\ell_1$ estimation error and the cumulative regret in Figure~\ref{fig:table_of_figures}.

%For each of the cases, we measure how $\|\hat{\theta}-\theta^*\|_1$ changes for each time $n$ (first row) and the bandit regret.
%\kj{plots: remove sigma, larger font. remove `unit vec'; remove the first column }

% \cite{hao2020high}
\noindent\textbf{Observation.} As we expected from the theoretical analysis, our estimator and bandit algorithm outperform the baselines.
In terms of the $\ell_1$ error, for both cases, we see that \ensuremath{\textsc{PopArt}} converges much faster than ${\Cmin}$-Lasso for large enough $n$.
Interestingly, $H^2$-Lasso also improves by just using the design computed for \ensuremath{\textsc{PopArt}} in case 1.
At the same time, $H^2$-Lasso is inferior than \ensuremath{\textsc{PopArt}} even if they are given the same data points.
While the design was optimized for \ensuremath{\textsc{PopArt}} and \ensuremath{\textsc{PopArt}} has the benefit of using the population covariance, which is unfair, it is still interesting to observe a significant gap between \ensuremath{\textsc{PopArt}} and Lasso.
For sparse linear bandit experiments, while ESTC requires exploration time almost the total length of the time horizon, ours requires a significantly shorter exploration phase in both cases and thus suffers much lower regret.

% % the two Lasso-based estimation methods in terms of the $\ell_1$ norm error.

% \kj{make the observation for each estimator separately. I think we had different intentions for both.}

\vspace{-8pt}
\section{Conclusion}
\label{sec:conclusion}
\vspace{-8pt}

We have proposed a novel estimator \ensuremath{\textsc{PopArt}} and experimental design for high-dimensional linear regression. %, which has enabled more accurate estimation with computational efficiency.
\ensuremath{\textsc{PopArt}} has not only enabled accurate estimation with computational efficiency but also led to improved sparse linear bandit algorithms.
Furthermore, we have closed the gap between the lower and upper regret bound on an important family of instances in the data-poor regime.

Our work opens up numerous future directions.
For \ensuremath{\textsc{PopArt}}, we speculate that $(Q(\mu)^{-1})_{ii}$ is the statistical limit for testing whether $\theta^*_i = 0$ or not -- it would be a valuable investigation to prove or disprove this.
We believe this will also help investigate whether the dependence on $H_*^2$ in our regret upper bound is unimprovable (note our matching lower bound is only for a particular family of instances).
Furthermore, it would be interesting to investigate whether we can use \ensuremath{\textsc{PopArt}} without relying on the population covariance; e.g., use estimated covariance from an extra set of unlabeled data or find ways to use the empirical covariance directly.
For sparse linear bandits, it would be interesting to develop an algorithm that achieves the data-poor regime optimal regret and data-rich regime optimal regret $\sqrt{sdn}$ simultaneously.
Furthermore, it would be interesting to extend our result to changing arm set, which poses a great challenge in planning.

\begin{ack}
We thank Ning Hao for helpful discussions on theoretical guarantees of Lasso.
Kwang-Sung Jun is supported by Data Science Academy and Research Innovation \& Impact at University of Arizona.
\end{ack}

% % \paragraph{Improving our result further}
% Even though we found out a matching upper and lower bound, there are still some problems worth studying, such as finding the real statistical limit of the $\ell_1$ norm error, relationship between $H_*^2$ and $\phi_0^2$, extension to the bandits with changing arm set, unified framework of the data-poor and data-rich regime in sparse linear bandits, or even the precise definition of the data-poor regime. For the perspective of the estimator, it would be a great improvement if \popart does not need $Q$ for the estimation, and use pseudo-inverse for the empirical covariance matrix instead.
% \paragraph{Other topics} There are interesting topics where one can apply the ideas in sparse linear bandit because of the structural similarity. We believe that low-rank bandit \cite{lu2021low} and compressed sensing \cite{zhang2014efficient} problems are the good examples of those fields.

% \red{
% future work
% \begin{itemize}
%     \item (?) low rank bandit extensions..
%     \item applications in compressed sensing?
%     \item prove the conjecture that $H^2$ is the statistical limit (cannot be improved) w.r.t. $\ell_1$ norm error. relationship with the compatibility constant.
%     \item (non-bandit point of view; purely about the estimator) semi-supervised learning: the idea is that you estimate $Q$ form a separate unlabeled data points and prove a bound for it.
%     \item the exact range of $n$ w.r.t. $s$, $d$, and $H^2$ for the data poor regime.
%     \item sparse linear bandit, unified optimality for both datapoor and datarich regime is $\min\{\sqrt{sdn}, (snH)^{2/3}\}$?
%     \item idea for future work: changing arm set without any assumptions / acceleration when the distribution satisfies certain conditions.
% \end{itemize}

% }

\putbib[library-shared]
\end{bibunit}
```

### Checklist

The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default to , , or . You are strongly encouraged to include a **justification to your answer**, either by referencing the appropriate section of your paper or providing a brief inline description. For example:

-   Did you include the license to the code and datasets?

-   Did you include the license to the code and datasets?

-   Did you include the license to the code and datasets?

Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

1.  For all authors...

    1.  Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

    2.  Did you describe the limitations of your work?

    3.  Did you discuss any potential negative societal impacts of your work?

    4.  Have you read the ethics review guidelines and ensured that your paper conforms to them?

2.  If you are including theoretical results...

    1.  Did you state the full set of assumptions of all theoretical results?

    2.  Did you include complete proofs of all theoretical results?

3.  If you ran experiments...

    1.  Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

    2.  Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

    3.  Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

    4.  Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

4.  If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

    1.  If your work uses existing assets, did you cite the creators?

    2.  Did you mention the license of the assets?

    3.  Did you include any new assets either in the supplemental material or as a URL?

    4.  Did you discuss whether and how consent was obtained from people whose data you're using/curating?

    5.  Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

5.  If you used crowdsourcing or conducted research with human subjects...

    1.  Did you include the full text of instructions given to participants and screenshots, if applicable?

    2.  Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

    3.  Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

# Appendix