SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Multi-layer Perceptron (MLP). Instead of deep convolution layers used in many neural networks designed for image classification task [7], GPN utilizes two simple perceptron layers with ReLU activation function as the encoding network, which maps high dimensional data to a latent space with a much smaller dimension, avoiding the curse of dimensionality for the density estimation on a (mapped) latent representation [25]. As each node is independent of the others in this step, the encoding map only considers the node features without any graph structure involved. Mathematically, the mapping can be expressed by
$$
\boldsymbol {z} _ {i} = f (\boldsymbol {x} _ {i}; \boldsymbol {\theta}) = W _ {2} \sigma \left(W _ {1} \boldsymbol {x} _ {i} + \mathbf {1} ^ {\intercal} \mathbf {b} _ {1}\right) + \mathbf {1} ^ {\intercal} \mathbf {b} _ {2},
$$
where  $\theta := \{W_1, W_2, \pmb{b}_1, \pmb{b}_2\}$  denotes a set of learning parameters. For simplicity we use the notation  $\mathbf{z}_i = f_{\theta}(\mathbf{x}_i)$ .
Normalizing Flow. Normalizing flow is used to estimate the density  $\mathbb{P}(\pmb{z}_i|k;\phi)$  for  $k\in [K]$  and learning parameters  $\phi$  as an invertible transformation  $q(\cdot ;k)$  of a base distribution, e.g. Normal distribution, which denotes the distribution of class  $k$  in the latent space. The default flow in GPN is the radial flow [27], given by
$$
q (\mathbf {z}; k) = \mathbf {z} + \frac {\beta (\mathbf {z} - \mathbf {z} _ {0})}{\gamma + \| \mathbf {z} - \mathbf {z} _ {0} \|}
$$
$$
\mathbb {P} \left(\mathbf {z} _ {i} \mid k; \phi\right) = p _ {z} \left(q ^ {- 1} \left(\mathbf {z} _ {i}; k\right)\right) | \det  \frac {\partial q ^ {- 1} (\cdot ; k)}{\partial \mathbf {z}}.
$$
where  $\mathbf{z}_0$  is a reference point and  $p_z(\cdot) \sim \mathcal{N}(0,1)$ . After estimating the density of the node  $i$  belonging to a specific class  $k$ , the pseudo evidence counts are scaled to the probability, i.e.,  $\beta_i^k \propto \mathbb{P}(\mathbf{z}_i|k;\phi)$ , GPN sets
$$
\beta_ {i} ^ {k} := g _ {\phi} \left(\boldsymbol {z} _ {i}\right) _ {k} = N _ {k} \cdot \mathbb {P} \left(\boldsymbol {z} _ {i} \mid k; \phi\right),
$$
where  $N_{k}$  is the number of training nodes that belong to the class  $k$ .
Personalized Page Rank. GPN applies a personalized page rank (PPR) module to diffuse the evidence among neighboring nodes. It is motivated by the work of Approximate Personalized Propagation of Neural Predictions (APPNP) [12] that is designed to decouple the prediction (only based on node features) with any encoding network and propagate with a personalized page rank (PPR) module (only based on edge information). In particular, PPR provides a personalized influence score matrix for each node that considers  $L$  hop of neighbors without involving any new parameters to learn and  $L$  is a hyperparameter:
$$
\boldsymbol {\beta} ^ {(l + 1)} = (1 - \gamma) \hat {A} \boldsymbol {\beta} ^ {(l)} + \gamma \boldsymbol {\beta} ^ {(0)},
$$
where  $\gamma$  is a hyper-parameter relating to the teleport probability,  $\hat{A}$  denotes the symmetrically normalized graph adjacency matrix with added self-loops (i.e.,  $\hat{A} \coloneqq D^{-1/2}AD^{-1/2}$  with the standard adjacency matrix  $A$ ), and  $l$  denotes the layer index with  $\beta^{(0)}$  obtained after the normalizing flow. The output of PPR is a set of concentration parameters, denoted by  $\alpha = h_{\gamma}(\beta^{(0)})$ .
Collectively for MLP, normalizing flow, and PPR, the network in GPN can be expressed by
$$
\boldsymbol {\alpha} _ {i} = 1 + h _ {\gamma} \left(g _ {\phi} \left(f _ {\theta} \left(\mathbf {x} _ {i}\right)\right)\right), \tag {15}
$$
for each node  $i$ , where the addition of 1 guarantees that the concentration parameter is strictly positive. In addition, an entropy regularization was considered by GPN defined by,
$$
\mathbb {H} (\operatorname {D i r} \left(\boldsymbol {\alpha} _ {i}\right)) = \log B \left(\boldsymbol {\alpha} _ {i}\right) + \left(\alpha_ {i 0} - K\right) \Psi \left(\alpha_ {i 0}\right) - \sum_ {k = 1} ^ {K} \left(\alpha_ {i k} - 1\right) \Psi \left(\alpha_ {i k}\right), \tag {16}
$$
where  $\alpha_{i0} = \sum_{k = 1}^{K}\alpha_{ik}$
Next, we provide two examples to describe oversight of [6, Theorem 1] and [32, Theorem 1] in the sense that both theorems assume an impossibility. Particularly the assumption is that a two-layer ReLU network can be represented by a set of affine mappings, each being full rank, from a finite set of regions to the latent space. However, we construct Example 9 and Example 10 to show this assumption is impossible.
Example 9. We start with a simple case where a two-layer ReLU network with input, hidden layer, and output of a scalar (1-dimensional) is considered for an easier interpretation of the results. One simple example of a two-layer ReLU network is expressed by
$$
z = f _ {\theta} (x) = 1 \cdot \sigma_ {R e L U} (1 \cdot x + 0) + 0. \tag {17}
$$
Following [15], we split the latent space into two affine regions, i.e.,
$$
z = \left\{ \begin{array}{l l} x & \text {i f} x \in [ 0, \infty) \\ 0 & \text {i f} x \in (- \infty , 0 ], \end{array} \right. \tag {18}
$$
labeled by  $Q^{(0)} = [0,\infty)$  and  $Q^{(1)} = (-\infty, 0]$ . We see the associated  $V^{(0)} = 1$  and  $V^{(1)} = 0$  in the affine representation (17) that certainly do not have independent rows, as required by [6, Theorem 1].
Example 10 extends the 1D case in Example 9 into a higher  $d$ -dimension, showing that there is always at least one affine region that produces a single value, i.e.  $f_{\theta}(Q^{(l)^*}) = \{\mathbf{v}\}$  when mapped into a ReLU network  $f_{\theta}$ .
Example 10. We consider the ReLU network,
$$
f _ {\theta} (\mathbf {x}) = C \sigma_ {R e L U} (B \mathbf {x}), \tag {19}
$$
where  $B, C \in \mathbb{R}^{d \times d}$  are matrices of full rank. Denote  $\mathbf{x}_j$  to be the solution to the equation,
$$
- \mathbf {e} _ {j} = B \mathbf {x}, \tag {20}
$$
where  $\mathbf{e}_j$  is the  $j$ th Euclidean standard basis. As  $B$  is assumed to be full rank, there is the unique solution of the corresponding  $\mathbf{x}_j$ .
Notice that the polytope,
$$
S = \left\{\sum_ {j = 1} ^ {d} a _ {j} \mathbf {x} _ {j} \mid a _ {j} \geq 0 \right\}, \tag {21}
$$
has non-zero measure in  $\mathbb{R}^d$ . Note that the ReLU network is constant by construction, as  $\sigma_{ReLU}(-\mathbf{e}_j) = \mathbf{0}$ . In other words, we have for  $x \in S$  that
$$
\mathbf {z} = f _ {\theta} (\mathbf {x}) = C \sigma_ {R e L U} (B \mathbf {x}) = C \mathbf {0} = \mathbf {0}. \tag {22}
$$
As in the previous example, the existence of  $S$  means that there exists some  $V^{(\cdot)} = \mathbb{0}_{d,d}$  which contradicts the assumption that all  $V$ 's have independent rows. Under this setting, the density does not approach zero, which is the conclusion of Theorem 1 in [32].
In this section, we provide the proof of all the theorems and corollaries. Note that until Theorem 8, We ignore the graph component  $h_{\gamma}$  and focus solely on the representational layer  $f_{\theta}$  and normalizing flow layer  $g_{\phi}$ .
Proof of Theorem 1 and Corollary 3. As  $f_{\theta}$  is arbitrary by assumption, we choose it in such a way that it maps a point in  $\mathcal{X}_k$  to a point inside the ball centered at  $\mathbf{z}_k$  with radius  $r_k$ , denoted by  $B(\mathbf{z}_k, r_k)$ ,
$$
f _ {\theta}: \mathcal {X} _ {k} \rightarrow \mathcal {Z} _ {k} \subset B \left(\mathbf {z} _ {k}, r _ {k}\right), \tag {23}
$$
where  $\mathbf{z}_k\in \mathcal{Z}$  with a minimal distance  $R$  between any two of them, i.e.,  $d(\mathbf{z}_k,\mathbf{z}_m) > R,\forall k,m\in [K]$ , and we define  $r > r_k,\forall k\in [K]$ . We then choose the normalizing flow to be,
$$
g _ {\phi} (\mathbf {z}; k) = 1 + N _ {k} \cdot \left\{ \begin{array}{l} \frac {1}{\operatorname {V o l} \left(B \left(\mathbf {z} _ {k} , r _ {k}\right)\right)}, \text {i f} \mathbf {z} \in B \left(\mathbf {z} _ {k}, r _ {k}\right), \\ 0, \text {o t h e r w i s e .} \end{array} \right. \tag {24}
$$
We add the value of 1 in the normalizing flow to produce valid evidence measures. We also assume that  $N_{k} = \mu (\mathcal{X}_{k}) > 0$ , where  $\mu$  is the Lebesgue measure function.
The global minimum of UCE occurs when UCE is equal to 0 for every class. Recall that
$$
\sum_ {k \in [ K ]} \operatorname {U C E} \left(g \left(\mathcal {Z} _ {k}\right), Y\right) = \sum_ {k \in [ K ]} \int_ {\mathcal {Z} _ {k}} \left(\Psi \left(\sum_ {m \in [ K ]} g _ {m} (\mathbf {z})\right) - \Psi \left(g _ {k} (\mathbf {z})\right)\right) d \mu . \tag {25}
$$
Using (23), we consider an upper bound of the right-hand side by integrating over the larger region, that is,
$$
\begin{array}{l} \sum_ {k \in [ K ]} \int_ {B \left(\mathbf {z} _ {k}, r _ {k}\right)} \left(\Psi \left(\sum_ {m \in [ K ]} g _ {m} (\mathbf {z})\right) - \Psi \left(g _ {k} (\mathbf {z})\right)\right) d \mu , (26) \\ = \sum_ {k \in [ K ]} \operatorname {V o l} \left(B \left(\mathbf {z} _ {k}, r _ {k}\right)\right) \cdot \left(\Psi \left(K + \frac {N _ {k}}{\operatorname {V o l} \left(B \left(\mathbf {z} _ {k} , r _ {k}\right)\right)}\right) - \Psi \left(1 + \frac {N _ {k}}{\operatorname {V o l} \left(B \left(\mathbf {z} _ {k} , r _ {k}\right)\right)}\right)\right). (27) \\ \end{array}
$$
According the recurrence relation of the digamma function:  $\Psi (x + 1) = \Psi (x) + 1 / x$ , we readily derive that,
$$
\sum_ {k \in [ K ]} \operatorname {U C E} (g (\mathcal {Z} _ {k}), Y) \leq \sum_ {k \in [ K ]} \operatorname {V o l} (B (\mathbf {z} _ {k}, r _ {k})) \cdot \sum_ {m = 1} ^ {K - 1} \left(K - m + \frac {N _ {k}}{\operatorname {V o l} (B (\mathbf {z} _ {k} , r _ {k}))}\right) ^ {- 1}. \tag {28}
$$
Taking the limit of the right-hand side yields
$$
\begin{array}{l} \lim  _ {r \rightarrow 0} \sum_ {k \in [ K ]} \operatorname {V o l} \left(B \left(\mathbf {z} _ {k}, r _ {k}\right)\right) \cdot \sum_ {m = 1} ^ {K - 1} \left(K - m + \frac {N _ {k}}{\operatorname {V o l} \left(B \left(\mathbf {z} _ {k} , r _ {k}\right)\right)}\right) ^ {- 1}, (29) \\ = \sum_ {k \in [ K ]} \lim  _ {r _ {k} \rightarrow 0} \operatorname {V o l} \left(B \left(\mathbf {z} _ {k}, r _ {k}\right)\right) \cdot \lim  _ {r _ {k} \rightarrow 0} \sum_ {m = 1} ^ {K - 1} \left(K - m + \frac {N _ {k}}{\operatorname {V o l} \left(B \left(\mathbf {z} _ {k} , r _ {k}\right)\right)}\right) ^ {- 1}. (30) \\ \end{array}
$$
It is straightforward for the following two limits to hold,
$$
\lim  _ {r _ {k} \rightarrow 0} \operatorname {V o l} (B (\mathbf {z} _ {k}, r _ {k})) = 0, \tag {31}
$$
$$
\lim  _ {r _ {k} \rightarrow 0} \sum_ {m = 1} ^ {K - 1} \left(K - m + \frac {N _ {k}}{\operatorname {V o l} \left(B \left(\mathbf {z} _ {k} , r _ {k}\right)\right)}\right) ^ {- 1} = 0, \tag {32}
$$
thus leading to
$$
\lim  _ {r \rightarrow 0} \sum_ {k \in [ K ]} \operatorname {V o l} \left(B \left(\mathbf {z} _ {k}, r _ {k}\right)\right) \cdot \sum_ {m = 1} ^ {K - 1} \left(K - m + \frac {N _ {k}}{\operatorname {V o l} \left(B \left(\mathbf {z} _ {k} , r _ {k}\right)\right)}\right) ^ {- 1} = 0. \tag {33}
$$
On the other hand, as  $\Psi (\sum_{k\in [K]}g_k(\mathbf{z})) - \Psi (g_k(\mathbf{z}))\geq 0$  , we have
$$
0 \leq \sum_ {k \in [ K ]} \int_ {\mathcal {Z} _ {k}} (\Psi \left(\sum_ {m \in [ K ]} g _ {m} (\mathbf {z})\right) - \Psi \left(g _ {k} (\mathbf {z})\right)) d \mu = \sum_ {k \in [ K ]} \operatorname {U C E} \left(g \left(\mathcal {Z} _ {k}\right), Y\right), \tag {34}
$$
which implies that  $\mathrm{UCE} \to 0$  as  $r \to 0$ . For  $r = 0$ , UCE is equal to zero, which leads to Corollary 3.
Proof of Theorem 4. We denote the parameters of  $g_{\phi}$  that represent the true analytic solution  $\hat{\phi} = \phi (\theta)$ . Note that this is a function with respect to the choice of  $\theta$ , that is, the true distribution is dependent on the representational mapping. In this proof, we focus on finding a value of  $\theta$  s.t.,
$$
\hat {\theta} = \arg \min  _ {\theta} \mathrm {U C E} (\alpha (\theta , \hat {\phi}), Y) = \arg \min  _ {\theta} \mathrm {U C E} (\alpha (\theta , \phi (\theta)), Y). \tag {35}
$$
As the true distribution is dependent on the representational mapping, we should consider a joint minimization problem with respect to both the representation map and density distribution.
We will separate the proof into two cases. Specifically, we prove Case 1 by contradiction, showing that if the set  $\mathcal{Z}_k$  mapped to by  $f_{\theta}$  from  $\mathcal{X}_k$  has a non-zero measure, then the global minimizer UCE = 0 can not be achieved. We then prove Case 2, under the assumption that a true analytical solution may achieve density evidence at a point, by showing that we may achieve the global minimizer on a point set.
Case 1: Non-Zero Measure. Suppose the true distribution on this set is a non-degenerate distribution. As the natural definitions of a probability distribution  $1 = \int_{\mathcal{Z}_k}d\mu$ , the UCE loss can be expressed by
$$
\sum_ {k \in [ K ]} \operatorname {U C E} \left(g \left(\mathcal {Z} _ {k}\right), Y\right) = \sum_ {k \in [ K ]} \int_ {\mathcal {Z} _ {k}} \left(\Psi \left(\sum_ {m \in [ K ]} g _ {m} (\mathbf {z})\right) - \Psi \left(g _ {k} (\mathbf {z})\right)\right) d \mu (\mathbf {z}). \tag {36}
$$
In order for the measure of  $\mathcal{Z}_k$  to have a density of  $\epsilon > 0$ , there exists a subset of  $\mathcal{Z}_k$  with non-zero measure  $\delta_k > 0$ , denoted  $\mathcal{Z}_k^*$ . Using similar techniques as the proof of Theorem 1 in reverse, we obtain,
$$
\sum_ {k \in [ K ]} \operatorname {U C E} \left(g \left(\mathcal {Z} _ {k}\right), Y\right) \geq \sum_ {k \in [ K ]} \int_ {\mathcal {Z} _ {k}} \left(\Psi \left(K + N _ {k} \epsilon\right) - \Psi \left(1 + N _ {k} \epsilon\right)\right) d \mu ,
$$
then for some  $k\in [K]$  there exists some  $\mathcal{Z}_k^*$
$$
\begin{array}{l} \sum_ {k \in [ K ]} \operatorname {U C E} \left(g \left(\mathcal {Z} _ {k}\right), Y\right) \geq \int_ {\mathcal {Z} _ {k} ^ {*}} \left(\Psi \left(K + N _ {k} \epsilon\right) - \Psi \left(1 + N _ {k} \epsilon\right)\right) d \mu , \\ = \delta_ {1} \cdot \left(\Psi \left(K + N _ {k} \epsilon\right) - \Psi \left(1 + N _ {k} \epsilon\right)\right). \\ \end{array}
$$
As  $\Psi$  is strictly increasing, then  $\delta_{2} = \Psi (K + N_{k}\epsilon) - \Psi (1 + N_{k}\epsilon) > 0$ , which implies that
$$
\sum_ {k \in [ K ]} \operatorname {U C E} \left(g \left(\mathcal {Z} _ {k}\right), Y\right) \geq \sum_ {k \in [ K ]} \delta_ {1} \cdot \delta_ {2} > 0.
$$
Therefore, we prove that if  $f_{\theta}$  maps to a measurable set, the UCE loss is necessarily non-zero.
Case 2: Zero Measure Sets. Corollary 3 shows that the zero UCE is achievable. If Case 1 fails, then we can conclude that only on a disjoint set  $\mathcal{Z}_k$  with measure 0 for each  $k$  is permissible to achieve the UCE to be 0. The exact choice of this set depends on the precise definitions of the probability distributions on a point set and their ability to achieve infinite densities. Here we constrain these possibilities by requiring the range of  $f_{\theta}$  to have non-zero measure or to be a point set if having zero measure<sup>2</sup>.
Proof of Theorem 6. Pick  $\mathbf{x} \in \mathcal{X}$  s.t.  $d(\mathbf{x}, \mathbf{x}_k) > \delta$  for  $x_k \in \mathcal{X}_k$  see that for any  $f_\theta$  where  $\theta \in \Gamma$  we have that  $\mathbf{x}$  is necessarily not mapped to  $\mathbf{z}_k \in \mathcal{Z}$  (if it were mapped in  $\mathcal{Z}$  then the preimage would contain it and thus we would have  $d(\mathbf{x}, \mathbf{x}_k) < \delta$ , which is a contradiction with our selection of  $x$ ). Recall that the density for any point mapped to  $z_k$  is infinite. That is the density of the associated points mapped to the point set is necessarily infinite and the density of points mapped elsewhere is necessarily smaller, namely 0, with the associated evidence 1. Our selected  $\mathbf{x}$  then has no evidence in favor of it belonging to a class  $k$  while any point in  $\mathbf{x}_k \in \mathcal{X}_k$  must have infinite evidence by our choice of a well-fit  $\theta$ .
Proof of Corollary 7. Notice that we can choose two types of such  $\theta$
Case 1 Let  $\theta$  for class  $k$  be chosen such that,
$$
f (\mathbf {x}) = \left\{ \begin{array}{l} \mathbf {z} _ {k} \text {i f} \mathbf {x} \in \mathcal {X} _ {k}, \\ 0 \text {o t h e r w i s e .} \end{array} \right. \tag {37}
$$
2We choose that the cardinality of the zero-measure set  $f_{\theta}(\mathcal{X}_k)$  to be finite (rather than countably infinite) as we do not want to detail precise topological arguments (like compactness and boundedness) about the pointsets and their respective preimages.
$$
f (x) = \left\{ \begin{array}{l} \mathbf {z} _ {k} \text {i f} d (\mathbf {x}, \hat {\mathbf {x}}) <   \delta \text {f o r a n y} \hat {x} \in \mathcal {X} _ {k}, \\ 0 \text {o t h e r w i s e .} \end{array} \right. \tag {38}
$$
If  $\mathbf{x} \in \mathcal{X}_k$  is mapped to  $\mathbf{z}_k$  then it is endowed with infinite density, moreover, it is believed to be an ID node belonging to class  $k$ . Thus, the nearby OOD being detected for these UCE minimizers is determined by arbitrary choice.
Proof of Theorem 8. First note that the ID nodes are mapped to have infinite evidence achieved at the points in the latent space  $\mathcal{Z}_k$ . As the representations of the OOD nodes are in  $\mathcal{Z}_k$  they are also endowed with infinite evidence. That is graph layers can only help separate nodes by pulling them towards the center of their own classes w.r.t. to the representation space this is only helpful if their representations are separate to begin with.
Lastly, we give a toy example showing heuristically that the proposed regularization yields a better separation of the OOD nodes from IDs, compared to the original GPN model without the distance-based regularization.
Example 11. Consider two ID classes (Class 1 and Class 2) and one OOD class with the following construction:
1. All nodes belonging to Class 1 have feature values sampled from  $\mathbf{x}^{(1)} = [1,0,0]$ .
2. All nodes belonging to Class 2 have feature values sampled from  $\mathbf{x}^{(2)} = [-1,0,0]$ .
3. All nodes belonging to the OOD class 2 have feature values sampled from  $\mathbf{x}^{(OOD)} = [0,1,v]$ .
4. We sample  $v$  from the uniform distribution  $U(-1, 1)$  independently for each sample in each class.
5. All nodes are connected to every node within their own class, leading to a graph of homophily 1.
6. Suppose the density function is true density distribution
7. Denote the PPR layer by  $\hat{h}$  that uses the right normalized adjacency matrix  $AD^{-1}$  rather than symmetrically normalized  $D^{-1/2}AD^{-1/2}$ , used in APPNP.
8. Suppose  $f_{\theta}$  is a linear function (i.e. no activation function) explicitly,  $W = \begin{bmatrix} W_{11} & W_{12} \\ W_{21} & W_{22} \\ W_{31} & W_{32} \end{bmatrix}$ .
Then GPN with our regularization can learn an embedding that makes it possible to separate classes 1, 2, and OOD nodes. Without regularization, OOD nodes lie between ID classes in the latent space.
Proof. A simple calculation for the project leads to
$$
\mathbf {z} _ {i} = [ X W ] _ {i} = \left\{ \begin{array}{l l} {[ W _ {1 1}, W _ {1 2} ]} & {\text {f o r c l a s s 1 n o d e s}} \\ {[ - W _ {1 1}, - W _ {1 2} ]} & {\text {f o r c l a s s 2 n o d e s}} \\ {[ W _ {2 1} + v W _ {3 1}, W _ {2 2} + v W _ {3 1} ]} & {\text {f o r O O D n o d e s}.} \end{array} \right. \tag {39}
$$
Clearly, the values of  $W_{31}$  and  $W_{32}$  would be smaller with the distance minimization term applied than without, as  $v$  is selected randomly. Moreover neither  $W_{31}$  nor  $W_{32}$  affects the model's ability to separate the two classes as desired. We explicitly calculate both UCE and the distance-based regularization in the objective function, while ignoring the Dirichlet regularization, thus leading to the following objective function,
$$
L (Z, \alpha , Y; \mathcal {G}) = \operatorname {U C E} (\hat {h} \left(g _ {\phi} \left(f _ {\theta} (x)\right)\right), Y) + R (Z; \mathcal {G}). \tag {40}
$$
First, we explicitly work out the distance-based regularization term
$$
\begin{array}{l} \mathrm {R} (Z; \mathcal {G}) = \sum_ {(i, j) \in \mathcal {E}} \| \mathbf {z} _ {i} - \mathbf {z} _ {j} \| ^ {2} \\ = \sum_ {(i, j) \in \mathcal {E} _ {1}} \| [ W _ {1 1}, W _ {1 2} ] - [ W _ {1 1}, W _ {1 2} ] \| ^ {2} + \sum_ {(i, j) \in \mathcal {E} _ {2}} \| [ - W _ {1 1}, - W _ {1 2} ] - [ - W _ {1, 1}, - W _ {1 2} ] \| ^ {2} \\ + \sum_ {(i, j) \in \mathcal {E} _ {\mathrm {O O D}}} \left\| [ W _ {2 1} + v ^ {(i)} W _ {3 1}, W _ {2 2} + v ^ {(i)} W _ {3 2} ] - [ W _ {2 1} + v ^ {(j)} W _ {3 1}, W _ {2 2} + v ^ {(j)} W _ {3 2} ] \right\| ^ {2} \\ = \sum_ {(i, j) \in \mathcal {E} _ {\mathrm {0 0 D}}} \left\| (v ^ {(i)} - v ^ {(j)}) [ W _ {3 1}, W _ {3 2} ] \right\| ^ {2}, \\ \end{array}
$$
which is minimized when  $W_{31}$ ,  $W_{32}$  go to zero.
Next, we consider the UCE loss portion. See that as we estimate the true density using  $g$  we will have no overlap between the two distributions  $\mathcal{Z}_1, \mathcal{Z}_2$ . We are left with  $W = \begin{bmatrix} W_{11} & W_{12} \\ W_{21} & W_{22} \\ 0 & 0 \end{bmatrix}$ ,
$$
Z _ {i} = \left[ X W \right] _ {i} = \left\{ \begin{array}{l} \left[ W _ {1 1}, W _ {1 2} \right] \text {f o r c l a s s 1 n o d e s} \\ \left[ - W _ {1 1}, - W _ {1 2} \right] \text {f o r c l a s s 2 n o d e s} \\ \left[ W _ {2 1}, W _ {2 2} \right] \text {f o r O O D n o d e s ,} \end{array} \right. \tag {41}
$$
If  $W$  is to remain full rank this will necessarily require either  $W_{21}$  or  $W_{22}$  to be non-zero. Thus OOD nodes will be mapped as we see in (41) to some distinct values - which can be separated after the application of APPNP as we expect APPNP to only average the values within each class.
D Additional Experimental Details
D.1 Descriptions of Baselines
Graph-based Kernel Dirichlet distribution Estimation (GKDE) [39]: Based on the high homophily property of most graphs (neighboring nodes tend to share the same class label), GKDE derives the evidence with the help of the node-level distances (shortest path in the graph) with training nodes belonging to the same class.
Label Propagation (LP) [32]: Following the idea of GKDE, LP collects the evidence by relying on the density of labeled nodes in neighborhoods rather than distance. An initial condition per class is defined and then a Personalized Page Rank is used as the diffusion.
VGCN-Energy [21]: It is a GCN-based model with energy score as the uncertainty estimation which maps each node to a single, non-probabilistic scalar called the energy. The energy score can be calculated as follows
$$
s _ {\text {e n e r g y}} ^ {i} = - T \log \sum_ {k = 1} ^ {K} \exp^ {\frac {\imath_ {\frac {k}{2}} ^ {i}}{T}},
$$
where  $l$  is the predicted logits of a neural network and temperature parameter  $T = 1$ .
GKDE-GCN [39]: GKDE-GCN utilizes a GCN network to estimate the multisource uncertainty by a Dirichlet distribution and then sample probability as well as the class prediction. The evidence derived from the aforementioned GKDE is as a teacher of concentration parameters of Dirichlet Distribution, and another deterministic GCN predicting the probability is used as a teacher for sampled probability. The overall loss is composed of the KL divergence between these two teachers with the corresponding distribution and Bayes risk with respect to the squared loss of sampled class prediction.
APPNP [12]: Given that message passing neural network suffers from the over-smoothing problem that limits the depth of the neural network, APPNP proposed to decouple the prediction and propagation where the prediction depends on the node features and propagation depends on interactions between nodes through edges. APPNP first uses any kind of neural network to embed the input space and diffuses information with a personalized page rank. For large graphs, they use power iteration to approximate a topic-sensitive page rank.
GPN [32]: GPN applies a normalizing flow to estimate the density of each class in the latent space embedded with an encoding network and then propagates the scaled density as the evidence.
D.2 Description of Datasets