Success Probability of the Babai Estimators for Box-Constrained Integer Linear Models

In many applications including communications, one may encounter a linear model where the parameter vector <inline-formula> <tex-math notation="LaTeX">$\hat { {x}}$ </tex-math></inline-formula> is an integer vector in a box. To estimate <inline-formula> <tex-math notation="LaTeX">$\hat { {x}}$ </tex-math></inline-formula>, a typical method is to solve a box-constrained integer least squares problem. However, due to its high complexity, the box-constrained Babai integer point <inline-formula> <tex-math notation="LaTeX">$ {x}^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula> is commonly used as a suboptimal solution. In this paper, we first derive formulas for the success probability <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula> of <inline-formula> <tex-math notation="LaTeX">$ {x}^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula> and the success probability <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {OB}}$ </tex-math></inline-formula> of the ordinary Babai integer point <inline-formula> <tex-math notation="LaTeX">$ {x}^ {\scriptscriptstyle \text {OB}}$ </tex-math></inline-formula> when <inline-formula> <tex-math notation="LaTeX">$\hat { {x}}$ </tex-math></inline-formula> is uniformly distributed over the constraint box. Some properties of <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {OB}}$ </tex-math></inline-formula> and the relationship between them are studied. Then, we investigate the effects of some column permutation strategies on <inline-formula> <tex-math notation="LaTeX">$ {P}^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula>. In addition to V-BLAST and SQRD, we also consider the permutation strategy involved in the LLL lattice reduction, to be referred to as LLL-P. On the one hand, we show that when the noise is relatively small, LLL-P always increases <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula> and argue why both V-BLAST and SQRD often increase <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula>; and on the other hand, we show that when the noise is relatively large, LLL-P always decreases <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula> and argue why both V-BLAST and SQRD often decrease <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula>. We also derive a column permutation invariant bound on <inline-formula> <tex-math notation="LaTeX">$P^ {\scriptscriptstyle \text {BB}}$ </tex-math></inline-formula>, which is an upper bound and a lower bound under these two opposite conditions, respectively. Numerical results demonstrate our findings. Finally, we consider a conjecture concerning <inline-formula> <tex-math notation="LaTeX">$ {x}^ {\scriptscriptstyle \text {OB}}$ </tex-math></inline-formula> proposed by Ma <italic>et al.</italic> We first construct an example to show that the conjecture does not hold in general, and then show that it does hold under some conditions.