Domain: Machine Learning & Artificial Intelligence

Can you prove that, for both (i) learning intervals on [0,1] under the uniform distribution and (ii) learning a linear separator in ℝᵈ under the uniform distribution on the unit sphere, any active learning algorithm requires Ω(d log(1/ε)) label requests to achieve error at most ε with probability ≥ 1/2 — and in particular, show that for each setting, there exists a distribution over target functions such that any algorithm making fewer than c·d log(1/ε) queries (for some constant c > 0) must output a hypothesis with error ≥ ε with probability at least 1/2?

Grading rubric:
criterion: Does the solution include a formula connecting query complexity k to packing size M (k ≥ log₂ M - 1)?
weight: 10
criterion: Does the solution explain that a deterministic algorithm with k queries can produce at most 2ᵏ distinct outputs?
weight: 10
criterion: Does the solution explain that each output hypothesis can cover at most one target in Θ?
weight: 10
criterion: Does the solution explicitly invoke Yao’s Minimax Principle?
weight: 10
criterion: Does the solution specify the grid spacing δ in terms of ε?
weight: 10
criterion: Does the solution show that the symmetric difference between any two intervals in Θ is ≥ δ > 2ε?
weight: 10
criterion: Does the solution apply the Packing Lemma to derive k = Ω(d log(1/ε)) for intervals?
weight: 10
criterion: Does the solution specify the angular separation γ ≈ ε needed to guarantee error > 2ε?
weight: 20
criterion: Does the solution conclude that any algorithm requires Ω(d log(1/ε)) queries for error ≤ ε with probability ≥ 1/2?
weight: 10

Answer:
### Theorem
For both settings—(i) learning intervals on $[0,1]$ and (ii) learning linear separators in $\mathbb{R}^d$—there exists a distribution over target functions such that any active learning algorithm (randomized or deterministic) requires $\Omega(d \log(1/\varepsilon))$ label requests to achieve error $\le \varepsilon$ with probability $\ge 1/2$.

---

### 1. General Framework: The Packing Lower Bound

To prove a lower bound for *any* algorithm (including randomized ones), we use **Yao’s Minimax Principle**. It implies that the worst-case error probability of a randomized algorithm is lower-bounded by the expected error probability of the best *deterministic* algorithm against a specific "hard" input distribution $\mathcal{P}$.

We construct $\mathcal{P}$ as a uniform distribution over a finite set of "packed" hypotheses.

**Lemma (The Packing Bound):**
Let $\Theta = \{h_1, \dots, h_M\}$ be a finite set of hypotheses such that for any distinct pair $h_i, h_j \in \Theta$, the **disagreement probability under the data distribution** is $d(h_i, h_j) = P_{x \sim \mathcal{D}}(h_i(x) \neq h_j(x)) > 2\varepsilon$.
If the target $h^*$ is chosen uniformly from $\Theta$, then any deterministic active learning algorithm making fewer than $\lfloor \log_2 M \rfloor - 1$ queries will output a hypothesis with error $> \varepsilon$ with probability $\ge 1/2$.

**Proof of Lemma:**
1.  **Disjoint Error Regions:** Since the pairwise disagreement probability between targets is $> 2\varepsilon$, the error balls $B(h_i, \varepsilon) = \{h \mid d(h, h_i) \le \varepsilon\}$ are disjoint.
2.  **Injectivity:** Consequently, any single hypothesis $\hat{h}$ output by the learner can fall into the $\varepsilon$-ball of **at most one** target in $\Theta$.
3.  **Decision Tree Capacity:** A deterministic algorithm making $k$ binary queries can be modeled as a decision tree with at most $2^k$ leaves. Let $\mathcal{L}$ be the set of hypotheses output at these leaves.
4.  **Counting:** The algorithm succeeds if the true target $h^*$ corresponds to a leaf that "covers" it. Since $|\mathcal{L}| \le 2^k$ and each leaf covers at most 1 target, the algorithm can succeed on at most $2^k$ of the $M$ possible targets.
5.  **Probability:** Under the uniform distribution over $\Theta$, the probability of success is at most $2^k / M$. For the error probability to be $< 1/2$, we require $2^k / M > 1/2$, or $k > \log_2 M - 1$.

---

### 2. Part (i): Intervals on $[0,1]$

**Setting:**
*   Instance space $X = [0,1]$, Uniform distribution.
*   Hypothesis class $\mathcal{H} = \{ [a, b] \mid 0 \le a \le b \le 1 \}$.
*   Problem dimension $d=2$ (two parameters).

**Proof:**
We invoke the Packing Lemma by constructing a set $\Theta$ of size $\approx (1/\varepsilon)^2$.

1.  **Construction:**
    Let $\delta = 3\varepsilon$. We define a grid of possible endpoints:
    *   Left endpoints $L = \{ 0, \delta, 2\delta, \dots \} \cap [0, 1/2]$.
    *   Right endpoints $R = \{ 1, 1-\delta, 1-2\delta, \dots \} \cap [1/2, 1]$.
    *   Let $\Theta = \{ [l, r] \mid l \in L, r \in R \}$.
    *   *Note:* The restriction to $[0, 1/2]$ and $[1/2, 1]$ ensures $l \le r$ for all pairs, making all intervals valid.

2.  **Separation:**
    For any two distinct intervals $h_1 = [l_1, r_1]$ and $h_2 = [l_2, r_2]$, the disagreement probability (measure of the symmetric difference) is:
    $$ d(h_1, h_2) = |l_1 - l_2| + |r_1 - r_2| $$
    Since $h_1 \neq h_2$, they must differ by at least one grid step $\delta$ in either the left or right coordinate. Thus, $d(h_1, h_2) \ge \delta = 3\varepsilon > 2\varepsilon$.

3.  **Cardinality ($M$):**
    The size of $L$ is $\approx \frac{1/2}{3\varepsilon} \propto \frac{1}{\varepsilon}$. Similarly, $|R| \propto \frac{1}{\varepsilon}$.
    $$ M = |L| \times |R| = \Omega\left(\frac{1}{\varepsilon^2}\right) $$

4.  **Lower Bound:**
    Applying the Lemma:
    $$ k \ge \log_2\left( \frac{c}{\varepsilon^2} \right) \approx 2 \log_2\left(\frac{1}{\varepsilon}\right) $$
    Since $d=2$, this confirms **$k = \Omega(d \log(1/\varepsilon))$**.

---

### 3. Part (ii): Linear Separators in $\mathbb{R}^d$

**Setting:**
*   Instance space $X = \mathbb{S}^{d-1}$ (unit sphere), Uniform surface measure.
*   Hypothesis class $\mathcal{H} = \{ x \mapsto \text{sign}(w \cdot x) \}$.
*   Problem dimension is $d$.

**Proof:**
We invoke the Packing Lemma by constructing a spherical code $\Theta$ of size roughly $(1/\varepsilon)^{d-1}$.

1.  **Geometry of Error:**
    For unit vectors $u, v$, the disagreement probability is exactly the normalized surface measure of the wedge between the hyperplanes defined by $u$ and $v$. This is proportional to the angle:
    $$ d(h_u, h_v) = \frac{\theta(u, v)}{\pi} $$
    To ensure separation $> 2\varepsilon$, we require angular separation $\theta > 2\pi\varepsilon$.

2.  **Construction (Spherical Packing):**
    We need a set $\Theta \subset \mathbb{S}^{d-1}$ such that every pair is separated by angle $\gamma \approx \varepsilon$. The maximum size of such a set is bounded by comparing the total surface area of the sphere to the area of a spherical cap of radius $\gamma/2$.
    Using standard bounds for spherical codes (e.g., Gilbert-Varshamov or Kabatiansky-Levenshtein), the packing number scales as:
    $$ M = \Omega\left( \left(\frac{1}{\gamma}\right)^{d-1} \right) = \Omega\left( \left(\frac{1}{\varepsilon}\right)^{d-1} \right) $$
    *(This holds up to constant factors in the exponent).*

3.  **Lower Bound:**
    Applying the Lemma:
    $$ k \ge \log_2 M - 1 \approx (d-1) \log_2 \left(\frac{1}{\varepsilon}\right) $$
    For $d \ge 2$, we have $d-1 \ge d/2$. Thus:
    $$ k = \Omega\left( d \log \frac{1}{\varepsilon} \right) $$

### Conclusion

In both cases, we established that the set of distinguishable hypotheses has a size $M$ that scales as $(1/\varepsilon)^{\Theta(d)}$. Since active learning queries provide at most 1 bit of information per query, identifying the correct target requires $\log_2 M$ queries.
Therefore, any algorithm requires $\Omega(d \log(1/\varepsilon))$ queries.