# Weak HCH accesses EXP

_This post is a follow-up to my “[Alignment proposals and complexity classes](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes)” post. Thanks to Sam Eisenstat for helping with part of the proof here._

[Previously](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes), I proved that [imitative amplification](https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#2__Imitative_amplification___intermittent_oversight) with [weak HCH](https://ai-alignment.com/humans-consulting-hch-f893f6051455), [approval-based amplification](https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#4__Approval_based_amplification___relaxed_adversarial_training), and [recursive reward modeling](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) access $\text{PSPACE}$ while [AI safety via market making](https://www.alignmentforum.org/posts/YWwzccGbcHMJMpT45/ai-safety-via-market-making) accesses $\text{EXP}$. At the time, I wasn't sure whether my market making proof would generalize to the others, so I just published it with the $\text{PSPACE}$ proofs instead. However, I have since become convinced that the proof does generalize—and that it generalizes for all of the proposals I mentioned—such that imitative amplification with weak HCH, approval-based amplification, and recursive reward modeling all actually access $\text{EXP}$. This post attempts to prove that.

# Updated list of proposals by complexity class

$\text{P}$: Imitation learning (trivial)

$\text{PSPACE}$: [AI safety via debate](https://openai.com/blog/debate/) ([proof](https://arxiv.org/pdf/1805.00899.pdf))

$\text{EXP}$: [AI safety via market making](https://www.alignmentforum.org/posts/YWwzccGbcHMJMpT45/ai-safety-via-market-making) ([proof](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes#AI_safety_via_market_making_accesses_EXP)), [Imitative amplification with weak HCH](https://ai-alignment.com/humans-consulting-hch-f893f6051455) (proof below), [Approval-based amplification](https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#4__Approval_based_amplification___relaxed_adversarial_training) (proof below), [Recursive reward modeling](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) (proof below)

$\text{NEXP}$: [Debate with cross-examination](https://www.alignmentforum.org/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1) ([proof](https://docs.google.com/document/d/1SsTeb0AEQvecckZjSmAxXJ-upcna5DxvD0s6YaQ14EA/edit?usp=sharing))

$\text{R}$: [Imitative amplification with strong HCH](https://ai-alignment.com/strong-hch-bedb0dc08d4e) ([proof](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes#Imitative_amplification_with_strong_HCH_accesses_R)), AI safety via market making with pointers ([proof](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes#AI_safety_via_market_making_with_pointers_accesses_R))

# Proofs

## Imitative amplification with weak HCH accesses $\text{EXP}$

The proof here is similar in structure to [my previous proof that weak HCH accesses $\text{PSPACE}$](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes#Imitative_amplification_with_weak_HCH_accesses_PSPACE), so I'll only explain where this proof differs from that one.

First, since $l \in \text{EXP}$, we know that for any $x \in X$, $T_l(x)$ halts in $O(2^{\text{poly}(n)})$ steps where $n = |x|$. Thus, we can construct a function $f_l(n) = c_1 + c_2 e^{c_3 n^{c_4}}$ such that for all $x \in X$, $T_l(x)$ halts in less than or equal to $f_l(x)$ steps by picking $c_3,~ c_4$ large enough that they dominate all other terms in the polynomial for all $n \in \mathbb N$. Note that $f_l$ is then computable in time polynomial in $n$.

Second, let $H$'s new strategy be as follows:

1. Given $p$, let $s,~ x = M(p : f(|x|))$. Then, return accept/reject based on whether $s$ is an accept or reject state (it will always be one or the other)
2. Given $p : 0$, return $s_0,~ x_0$ where $s_0$ is the starting state and $x_0$ is the empty tape symbol.
3. Given $p : i$, let
\begin{align*}
& s,~ x = M(p : i - 1) \\
& x_{\text{left}} = M(p : i - 1 : -1) \\
& x_{\text{right}} = M(p : i - 1 : 1).
\end{align*}
Then, return $s^\prime,~ x^\prime$ where $s^\prime$ is the next state of $T_l$ from state $s$ over symbol $x$ (where if we're already in an accept/reject state we just stay there) and $x^\prime$ is the new tape symbol at the head (which can be determined given $s,~ x,~ x_\text{left},~ x_\text{right}$).
4. Given $p : 0 : j$, return $x_0$.
5. Given $p : i : j$, let $s,~ x_\text{head} = M(p : i - 1)$. Then, let $h$ be the amount by which $T_l$ on $s,~ x_\text{head}$ changes the head position by (such that $h \in \{-1, 0, 1\}$). If $j + h = 0$, let $s^\prime,~ x^\prime = M(p : i - 1)$ otherwise let $x^\prime = M(p : i - 1 : j + h)$. Return $x^\prime$.

Note that this strategy precisely replicates the strategy used in [my proof that market making accesses $\text{EXP}$](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes#AI_safety_via_market_making_accesses_EXP) for inputs $p : i$ and $p : i : j$. Thus, I'll just defer to that proof for why the strategy works and is polynomial time on those inputs. Note that this is where $l \in \text{EXP}$ becomes essential, as it guarantees that $i$ and $j$ can be represented in polynomial space.

Then, the only difference between this strategy and the market making one is for the base $p$ input. On $p$, given that $M(p : i)$ works, $M(p : f(|x|))$ will always return a state after $T_l$ has halted, which will always be an accept state if $T_l$ accepts and a reject state if it rejects. Furthermore, since $|M(p : f(|x|)| \in O(1)$ and $f$ is computable in polynomial time, the  strategy for $p$ is polynomial, as desired.

Since this procedure works for all $l \in \text{EXP}$, we get that amplification with weak HCH accesses $\text{EXP}$, as desired.

## Approval-based amplification accesses $\text{EXP}$

The proof here is almost precisely the same as [my previous proof that approval-based amplification accesses $\text{PSPACE}$](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes#Approval_based_amplification_accesses_PSPACE) with the only modification being that we need to verify that $|M(q)| + |H(q, M)|$ are still polynomial in size for the new imitative amplification strategy given above. However, [the fact that $i$ and $j$ are bounded above by $t_p$](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes#AI_safety_via_market_making_accesses_EXP) means that they are representable in polynomial space, as desired.

## Recursive reward modeling accesses $\text{EXP}$

Again, the proof here is almost precisely the same as [my previous proof that recursive reward modeling accesses $\text{PSPACE}$](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes#Recursive_reward_modeling_accesses_PSPACE) with the only modification being the same as with the above proof that approval-based amplification accesses $\text{EXP}$.