SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daesol Cho, Seungjae Lee, and H. Jin Kim
dscho1234@snu.ac.kr, ysz0301@snu.ac.kr, hjinkim@snu.ac.kr
Reinforcement learning (RL) often faces the challenges of uninformed search problems where the agent should explore without access to the domain knowledge such as characteristics of the environment or external rewards. To tackle these challenges, this work proposes a new approach for curriculum RL called Diversify for Disagreement & Conquer (D2C). Unlike previous curriculum learning methods, D2C requires only a few examples of desired outcomes and works in any environment, regardless of its geometry or the distribution of the desired outcome examples. The proposed method performs diversification of the goal-conditional classifiers to identify similarities between visited and desired outcome states and ensures that the classifiers disagree on states from out-of-distribution, which enables quantifying the unexplored region and designing an arbitrary goal-conditioned intrinsic reward signal in a simple and intuitive way. The proposed method then employs bipartite matching to define a curriculum learning objective that produces a sequence of well-adjusted intermediate goals, which enable the agent to automatically explore and conquer the unexplored region. We present experimental results demonstrating that D2C outperforms prior curriculum RL methods in both quantitative and qualitative aspects, even with the arbitrarily distributed desired outcome examples.
Reinforcement learning (RL) has great potential for the automated learning of behaviors, but the process of learning individual useful behavior can be time-consuming due to the significant amount of experience required for the agent. Furthermore, in its general usage, RL often involves solving a challenging uninformed search problem, where informative rewards or desired behaviors are rarely observed. While there are some techniques that can alleviate the exploration burden such as reward-shaping [30] or preference-based reward [21, 6], they often require significant domain knowledge or human intervention. This makes it difficult to utilize RL directly for problems that require challenging exploration, especially in many domains of practical significance. Thus, it is becoming increasingly crucial to tackle these challenges from the algorithmic level by developing RL agents that can learn autonomously with minimal supervision.
One potential approach is a curriculum learning algorithm. It involves proposing a carefully designed sequence of curriculum tasks or goals for the agent to accomplish, with each step building upon the previous one to gradually progress the curriculum. By doing so, it enables the agent to automatically explore the environment and improve the capability in a structured way. Previous studies primarily involve a mechanism to adjust the curriculum distribution by maximizing its entropy within the explored region [35], or taking into account the difficulty level [43, 11], or learning progress of the agent [36]. However, efficient desired outcome-directed exploration is not available under these
Figure 1: D2C trains a set of classifiers with labeled source data while diversifying their outputs on unlabeled target data (red: predicted label 0, blue: predicted label 1). Then, it proposes curriculum goals based on the diversified classifier's disagreement and similarity-to-desired outcome.
frameworks as these approaches do not have converging curriculum objectives, resulting in a naive search for unseen states.
Providing the agent with examples of successful outcomes can make the RL problem more manageable, as opposed to letting the agent explore without a particular objective and with an undirected reward. Such examples can offer considerable guidance on how to achieve a task if the agent can estimate the similarities between the desired outcome examples and visited states. In the realm of desired outcome-directed RL, previous studies try to maximize the probability of reaching desired outcomes [12, 41, 8]. However, these approaches do not have tools for explicitly quantifying an unexplored region, leading the agent to settle for reaching an initially discovered desired outcome example, rather than continuing to explore further to find another example. Other works try to minimize the distance between the generated curriculum distribution and the desired outcome distribution to propose intermediate task goals [37, 18]. But, these approaches have mainly been confined to problems that do not involve significant exploration challenges because they rely on the assumption that the Euclidean distance metric can represent the geodesic interpolation between distributions. This assumption is not universally applicable to all environmental geometries, making these methods less versatile than desired.
Therefore, it is necessary to develop an algorithm that enables the agent to automatically perform the outcome-directed exploration by generating a sequence of curriculum goals, which can be applied to arbitrary geometry and distribution of the given desired outcome examples. To do so, we propose Diversify for Disagreement & Conquer (D2C), which only requires desired outcome examples and does not require prior domain knowledge such as 1) the geometry of the environment 2) or the number of modes or distribution of the desired outcomes, or 3) external reward from the environment. Specifically, D2C involves diversifying the goal-conditioned classifiers to identify the similarities between the visited states and the desired outcome states. This is accomplished by ensuring that the outputs of each classifier disagree on unseen states, which not only allows determining the unexplored frontier region but also provides an arbitrary goal-conditioned intrinsic reward. Based on such conditional classifiers, we propose to employ bipartite matching to define a straightforward and easy-to-comprehend curriculum learning objective, which produces a range of well-adjusted curriculum goals that interpolate between the initial state distribution and arbitrarily distributed desired outcome states for enabling the agent to conquer the unexplored region.
To sum up, our work makes the following key contributions.
- We propose a new outcome-directed curriculum RL method that only needs a few arbitrarily distributed desired outcome examples and eliminates the need for external reward.
- To the best of our knowledge, D2C is the first algorithm for curriculum RL that allows for automatic progress in any environment without being restricted by its geometry or the desired outcome distribution by proposing diversified conditional classifiers.
- In various goal-conditioned RL experiments, our method consistently outperforms the previous curriculum RL methods through precisely calibrated guidance toward the desired outcome states in both quantitative and qualitative aspects.
Despite various attempts to improve exploration in RL, it remains a difficult issue that has not been fully solved. Some previous works for exploration have suggested methods based on information theoretical approaches [9, 39, 50, 20, 17, 27], maximizing the state visitation distribution's entropy [47, 25, 26], counting the state visitation [3, 31], utilizing similarity or curiosity [33, 45], and quantifying uncertainty through prediction models [4, 34]. Other approaches provide a curriculum to allow the agent to explore the environment through intermediate tasks. Curricula are usually created by adjusting the distribution of goals to cover new and unexplored areas. It is achieved by considering an auxiliary objective such as entropy [35] or disagreement between the model ensembles [49, 15, 28] or difficulty level of the curriculum [11, 43], regret [16], and learning progress [36]. However, these methods only focus on visiting diverse frontier states or do not provide a mechanism to converge toward the desired outcome distribution. In contrast, our method enables more efficient outcome-directed exploration via curriculum proposal with an objective to converge rather than simply exploring various frontier states, only requiring a few desired outcome examples.
Assuming access to desired outcome samples or distribution, some prior methods try to accomplish the desired outcome states by maximizing the probability of reaching these states [12, 41, 8]. However, they lack a mechanism for quantifying an under-explored region and synthesizing the knowledge acquired from the agent's experiences into versatile policies that can accomplish novel test goals. Some algorithms generate curricula as an interpolation between the distribution of desired target tasks and auxiliary tasks [37, 18], but they still rely on the Euclidean distance metric, which is insufficient to handle arbitrary geometric structures. There exists a work that addresses geometry-agnostic curriculum generation using the given desired outcome states, similar to our approach [5]. But, it requires carefully tuned Wasserstein distance estimation that depends on the optimality of the agent. It leads to inconsistent estimation before the convergence, resulting in numerically unstable training, while our method does not have such dependence. Also, it adopts a meta-learning-based technique [10, 23] that requires gradient computation at every optimization iteration, while our method only requires a single neural network inference, leading to much faster curriculum optimization.
A core idea behind our method is utilizing classifiers to learn a diverse set of hypotheses that minimize the loss on source inputs but make differing predictions on target inputs [32, 22]. It is related to ensemble methods [7, 19, 14] that aggregate the multiple functions' predictions, but the proposed method is distinct in terms of directly optimizing on an underspecified target dataset for enhancing diversity. Even though this diversifying strategy for target data is typically considered from the perspective of out-of-distribution robustness in conditions of distribution shift [29, 24, 38] or domain adaptation [44, 46, 42], we demonstrate how it can be applied for classifiers to quantify the similarity between the visited states and desired outcome states, and discuss its links to exploration and curriculum generation in RL.
We consider the Markov decision process (MDP)  $\mathcal{M} = (\mathcal{S},\mathcal{G},\mathcal{A},\mathcal{P},\gamma)$ , where  $\mathcal{S}$  indicates the state space,  $\mathcal{G}$  the goal space,  $\mathcal{A}$  the action space,  $\mathcal{P}(s'|s,a)$  the transition dynamics, and  $\gamma$  the discount factor. In our framework, the MDP is not provided a reward function and we consider a setting where only the desired outcome examples  $\{g_k^+\}_{k=1}^K$  from the desired outcome distribution  $p^+(g)$  are given. Thus, our method utilizes an intrinsic reward  $r: \mathcal{S} \times \mathcal{G} \times \mathcal{A} \to \mathbb{R}$ . Also, we represent the curriculum distribution obtained by our method as  $p^c(s)$ .
3.1 Acquiring knowledge from underspecified data
For diversification-based curriculum RL, we train a classification model  $y = f(x)$  in a supervised learning setting where  $x \in \mathcal{X}$  is input, and  $y \in \mathcal{Y}$  is the corresponding label. The model  $f$  is trained with a labeled source dataset  $\mathcal{D}_{\mathrm{S}} \sim p_{\mathrm{S}}(x,y)$  that includes both the given desired outcome examples  $(y = 1)$  and the visited states in the replay buffer  $\mathcal{B}$  of the RL agent  $(y = 0)$ . We assume that the desired outcome distribution can be modeled as a mixture of outcome distribution,  $p^{+}(x) = \sum_{o \in \mathbb{O}} w_{o} p_{o}(x)$ , where each  $o \in \mathbb{O}$  corresponds to a specific outcome distribution  $p_{o}(x)$ . The selection of model  $f$  from hypothesis class  $f \in \mathcal{F}$  is achieved by minimizing the predictive risk  $\mathbb{E}_{p_{\mathrm{S}}(x,y)}[\mathcal{L}(f(x),y)]$ , where  $\mathcal{L}$  is a standard classification loss.
Although  $f$  demonstrates good generalization on previously unseen data acquired from the source distribution  $p_{\mathrm{S}}(x,y)$ , it is ambiguous to evaluate the model  $f$  in distribution shift conditions such as when querying target data obtained from an out-of-distribution (e.g. a state lies in an unexplored region). This is because there could be many potential models  $f$  that can minimize the predictive risk. To formalize this intuition, we introduce the concept of an  $\varepsilon$ -optimal set defined as  $\mathcal{F}^{\varepsilon} \coloneqq \{f \in \mathcal{F} \mid \mathcal{L}_p(f) \leq \varepsilon\}$  [22], where  $\mathcal{L}_p$  represents the risk associated with a distribution  $p$ , and  $\varepsilon \geq 0$ .
The definition of the  $\varepsilon$ -optimal set implies that the predictions of any two models  $f_{1}, f_{2}$  in the set are almost identical on  $p_{\mathrm{S}}(x, y)$  when  $\varepsilon$  is small. But, using  $\mathcal{F}^{\varepsilon}$  as it is has a few drawbacks: 1) there is no criterion to prefer any particular hypotheses of  $\mathcal{F}^{\varepsilon}$  over another, and 2) the trained model  $f$  might not be suitable for evaluating the target data distribution as it does not have a mechanism to quantitatively discriminate unseen target data from the labeled source data.
To address this point, we utilize an unlabeled target dataset  $\mathcal{D}_{\mathrm{T}} \sim p_{\mathrm{T}}(x)$  for comparing the functions within  $\mathcal{F}^{\varepsilon}$  by examining how their predictions differ on  $\mathcal{D}_{\mathrm{T}}$ . Since  $\mathcal{D}_{\mathrm{T}}$  can be viewed as indicating the directions of functional change that are most crucial to the automatic exploration of the RL agent, we set  $p_{\mathrm{T}}(x)$  as the uniform distribution between the lower and upper bound of the state space to include all possible candidate states to visit. Knowing the state space's bounds is a commonly utilized assumption in many curriculum learning works [37, 18, 5] and it does not require being aware of the dynamically feasible areas. Then, our objective is to identify a set of classification models that perform well in  $p_{\mathrm{S}}(x,y)$  and disagree in  $p_{\mathrm{T}}(x)$  to recognize the unseen state from the unexplored region by quantifying the similarity between the observed states in  $\mathcal{B}$  and desired outcome examples. Such a function will lie within  $\mathcal{F}^{\varepsilon}$  of  $p_{\mathrm{S}}(x,y)$ , and the model leverages  $\mathcal{D}_{\mathrm{T}}$  to identify a diverse set of functions within the near-optimal set.
For an automatic exploration toward the desired outcome distribution via calibrated guidance of the curriculum, the proposed D2C suggests curriculum goals via diversified classifiers that disagree in the unexplored region and enables the agent to conquer this area by exploring through the goal-conditioned shaped intrinsic reward. It allows the agent to make progress without any prior domain knowledge of the environment such as obstacles or distribution of the desired outcome states and advances the curriculum towards the desired outcome distribution  $p^{+}(g)$ .
4.1 Diversification for disagreement on underspecified data
For the automatic exploration of the RL agent with the curriculum proposal, quantification of whether a queried state is already explored or not is required. To obtain such a quantification, we diversify a collection of classifier functions by comparing predictions for the target dataset while minimizing the training error as briefly described in Section 3.1. The intuition behind this is that diversifying predictions will produce functions that disagree with data in the ambiguous region [22].
Specifically, we use a multi-headed neural network with  $N$  heads to train diverse multiple functions. Each head  $i$  produces a prediction for an input  $x$  represented as  $f_{i}(x)$ . To ensure that every head has a low predictive risk on  $p_{\mathrm{S}}(x,y)$ , we minimize the cross-entropy loss for each head using  $\mathcal{L}_{\mathrm{xent}}(f_i) = \mathbb{E}_{x,y\sim \mathcal{D}_{\mathrm{S}}}\left[\mathcal{C}\mathcal{E}(f_i(x),y)\right]$ . Ideally, it is desirable for each function to rely on distinctive predictive features of the input to encourage differing predictions. Therefore, we train each pair of heads to generate statistically independent predictions, which implies disagreement in predictions. It could be achieved by minimizing the mutual information between each pair of heads:
$$
\mathcal {L} _ {\mathrm {M I}} \left(f _ {i}, f _ {j}\right) = \mathbb {E} _ {x \sim \mathcal {D} _ {\mathrm {T}}} \left[ D _ {\mathrm {K L}} \left(p \left(f _ {i} (x), f _ {j} (x)\right) \| p \left(f _ {i} (x)\right) \otimes p \left(f _ {j} (x)\right)\right) \right] \tag {1}
$$
where the input data is obtained from target dataset  $\mathcal{D}_{\mathrm{T}}$ . For implementation, we compute empirical estimates of the joint distribution  $p(f_{i}(x),f_{j}(x))$  and the product of the marginal distributions  $p(f_{i}(x))\otimes p(f_{j}(x))$ , which can be computed using libraries developed for deep learning [22].
In summary, the overall objective for classifier diversification is represented with a hyperparameter  $\lambda$ :
$$
\sum_ {i} \mathcal {L} _ {\text {x e n t}} \left(f _ {i}\right) + \lambda \sum_ {i \neq j} \mathcal {L} _ {\mathrm {M I}} \left(f _ {i}, f _ {j}\right) \tag {2}
$$
Figure 2: Overview of the curriculum proposal when we select two candidates for curriculum goals of  $g_{i}^{+}$  by bipartite matching. (1) The curriculum goals are proposed according to the quantification of the similarity-to-desired outcome ( $s_{i}$  with a high probability will be selected). (2) If the classifier is not in the conditional form, it cannot distinguish the source of the desired outcome example  $g_{i}^{+}$ , resulting in collapsed curriculum proposals, (3) while the conditional classifier enables non-collapseded curriculum proposals even when the agent achieves a specific desired outcome (e.g.  $g_{2}^{+}$ ) first.
(1)
p(y=1|s)	s1	s2	s3	s4	s5
g1+	0.5	0.4	1	0.9	0
g2+	0.5	0.4	1	0.9	0
p(y=1|s;g+)	s1	s2	s3	s4	s5
g1+	0.5	0.4	0	0	0
g2+	0.5	0.4	1	0.9	0
4.2 Quantifying unexplored regions by conditional classifiers
From this section, we slightly abuse the notation  $s$  instead of  $x$  for the RL setting. Considering the diversification process in Section 4.1, we can quantify how much a queried state  $s$  is similar to the desired outcome example from  $p^+(g)$ . Specifically, we can define a pseudo probability of the queried point  $s$  by averaging predictions of each head:
$$
p _ {\text {p s e u d o}} (y = 1 | s) := \frac {1}{N} \sum_ {i = 1} ^ {N} f _ {i} (s) \tag {3}
$$
When the queried state  $s$  is in proximity to the data within  $p_{\mathrm{S}}$  (with the desired outcome example labeled as 1 and states from the replay buffer  $\mathcal{B}$  labeled as 0), it becomes challenging for the classifiers to give a high probability to labels that substantially differ from the neighboring data, since each classifier  $f_{i}$  is trained to minimize the loss for the source data. On the other hand, if the queried state  $s$  is significantly different from the data in  $p_{\mathrm{S}}$ , each prediction head will output different values since the classifiers are trained to diversify the predictions on the target data (uniform distribution on the state space) to make their prediction values disagree as much as possible. For example, in the case of two heads, one predicts 1 and the other predicts 0 for the same queried target data, resulting in the pseudo probability of 0.5. This could be interpreted as a quantification of how much disagreement exists between classifiers, or uncertainty of the queried data, which can be utilized as an unexplored region-aware classification (Figure 1).
Thus, we can consider a curriculum learning objective for interpolating the curriculum distribution from the initial state distribution to  $p^{+}(g)$  represented by the following cross-entropy loss:
$$
\mathcal {L} _ {\text {c u r r}} = \mathbb {E} _ {s \sim \mathcal {B}, g ^ {+} \sim p ^ {+} (g)} \left[ \mathcal {C E} \left(p _ {\text {p s e u d o}} (y = 1 | s); y = p _ {\text {p s e u d o}} (y = 1 | g ^ {+})\right) \right] \tag {4}
$$
Intuitively, before discovering the desired outcome examples in  $p^+(g)$ , the curriculum goal candidate  $s \sim \mathcal{B}$  that minimizes Eq (4) is proposed in the frontier of the explored regions where classifiers  $f_i$  disagree. And, as the agent explores and discovers the desired outcome examples, the curriculum candidate is updated to converge to  $p^+(g)$  to minimize the discrepancy between the predicted labels of  $g^+$  and  $s$ .
However, it is not applicable for a case when the desired outcome examples are spread over multimodal distribution or arbitrarily because the trained classifiers  $f_{i}$  do not distinguish which  $p_{o}$  the desired outcome example is obtained from (Figure 2). In other words,  $f_{i}$  will predict a value close to
1 for any desired outcome example  $g^{+}$ , and the loss in Eq (4) will be close to 0. As the curriculum goal candidates are obtained from the replay buffer  $\mathcal{B}$ , the curriculum optimization may collapse if the agent achieves one of the desired outcome distributions  $(p_o)$  earlier, which is not desirable for achieving all the desired outcome examples regardless of its distribution.
Since we assume that we do not know which  $p_{o}$  the desired outcome example is obtained from, nor the number of modes  $(o)$  of the desired outcome distributions, we propose to utilize a conditional classifier to address this point while satisfying the assumption. Specifically, we define goal-conditioned classifiers, where each classifier takes input  $s$  and is conditioned on  $g$ , and these classifiers are trained to minimize the following modified objective of Eq (2):
$$
\begin{array}{l} \mathbb {E} _ {g \sim \mathcal {D} _ {\mathrm {G}}} \left[ \mathbb {E} _ {s \sim \mathcal {B}} \left[ \sum_ {i} \mathcal {L} _ {\text {x e n t}} \left(f _ {i} (s; g), y = 0\right) \right] + \mathbb {E} _ {\varepsilon} \left[ \sum_ {i} \mathcal {L} _ {\text {x e n t}} \left(f _ {i} (g + \varepsilon ; g), y = 1\right) \right] \right. \tag {5} \\ \left. + \lambda \mathbb {E} _ {s \sim \mathcal {D} _ {\tau}} \left[ \sum_ {i \neq j} \mathcal {L} _ {\mathrm {M I}} \left(f _ {i} (s; g), f _ {j} (s; g)\right) \right] \right] \\ \end{array}
$$
where  $\varepsilon$  is small noise (from uniform distribution around zero or standard normal distribution with a small variance) for numerical stability, and  $\mathcal{D}_{\mathrm{G}}$  can be either  $\mathcal{D}_{\mathrm{T}}$  or  $\mathcal{B} \cup p^{+}(g)$  for training arbitrary goal-conditioned & unexplored region-aware classifiers. We found that there is no significant difference between these choices. Then, the pseudo probability can be represented as  $p_{\mathrm{pseudo}}(y = 1|s; g) \coloneqq \frac{1}{N} \sum_{i=1}^{N} f_i(s; g)$  and we can address the arbitrarily distributed desired outcome examples without curriculum collapse (Figure 2). This proposed conditional classifier-based quantification is one of the key differences from the previous similar outcome-directed RL methods [23, 5]. Because the previous works require computing gradients of thousands of data for meta-learning-based network inference, while our method only requires a single feedforward inference without backpropagation which leads to fast computation.
4.3 Curriculum optimization via bipartite matching
As we assume that we have access to desired outcome examples from  $p^+(g)$  instead of their explicit distribution, we can approximate it using the sampled set  $\hat{p}^+(g)$  ( $|\hat{p}^+(g)| = K$ ). Then, the problem is formulated by the combinatorial setting that requires finding the curriculum goal candidate set  $\hat{p}^c(s)$  that will be assigned to each sample of  $\hat{p}^+(g)$ , and it can be solved via bipartite matching. With curriculum goal candidates and desired outcome examples, the curriculum learning objective is represented as follows: