SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The resultant noises mislead the model training, resulting in a terrible performance reduction. Our uncertainty-aware approach can mitigate the detriment caused by these data and improve the model robustness effectively.
E Proof of Ambiguous Data with High Information Entropy
In Section 1, we propose a theorem that "the ambiguous multi-modal data highly associated with multiple semantics, such as fast-paced videos and non-detailed texts, lead to high information entropy, standing for high uncertainty." The mathematical Theorem and Proof are given below.
Theorem. Suppose  $z \in \mathbb{R}^D$  is the vector of multi-modal data. Assuming the modality semantic subspace can be divided into  $K$  finite categories as  $X = \{x_i \in \mathbb{R}^D : i = 1, 2, \dots, K\}$ . The information entropy is denoted as Eq. 13:
$$
\mathcal {H} (X) = - \sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log p \left(x _ {i}\right) \quad \text {s . t .} \quad \sum_ {i = 1} ^ {K} p \left(x _ {i}\right) = 1, \tag {13}
$$
where  $p(x_{i})$  is the  $i^{th}$  semantics probability derived from cosine similarity  $\cos (z,x_i)$  through a softmax layer. Let  $Q\sim q(x)$  be a Discrete Uniform Distribution [14], then we have:
$$
\mathcal {J} (P \| Q) \propto \frac {1}{\mathcal {H} (X)}, \tag {14}
$$
where  $\mathcal{J}(P\parallel Q)$  denotes the Jensen-Shannon Divergence (JSD) [16], a metric to measure the distance between two probability distributions. Namely, Eq. 14 means that the closer the distribution  $P\sim p(x)$  is to the Uniform Distribution  $Q\sim q(x)$ , the higher the information entropy  $\mathcal{H}(X)$  is.
Proof. Let's first figure out what distribution of information entropy is the largest. Given an objective function (Eq. 15) and its constraint (Eq. 13):
$$
\max  _ {P} \mathcal {H} (X) = - p \left(x _ {1}\right) \log p \left(x _ {1}\right) - p \left(x _ {2}\right) \log p \left(x _ {2}\right) - \dots - p \left(x _ {K}\right) \log p \left(x _ {K}\right). \tag {15}
$$
Let  $\lambda$  denote Lagrange multiplier. The Lagrangian function can be constructed as:
$$
\ell (p, \lambda) = - \sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log p \left(x _ {i}\right) + \lambda \left(\sum_ {i = 1} ^ {K} p \left(x _ {i}\right) - 1\right). \tag {16}
$$
Separately computing the partial derivatives for  $p(x_{1}), p(x_{2}), \ldots, p(x_{K})$ :
$$
\frac {\partial \ell (p , \lambda)}{\partial p \left(x _ {i}\right)} = \lambda - \log p \left(x _ {i}\right) = 0. \tag {17}
$$
From this:
$$
\log p \left(x _ {1}\right) = \log p \left(x _ {2}\right) = \dots = \log p \left(x _ {K}\right) = \lambda . \tag {18}
$$
Considering the constraints of Eq. 13, the  $p(x)$  essentially satisfy:
$$
p \left(x _ {1}\right) = p \left(x _ {2}\right) = \dots = p \left(x _ {K}\right) = \frac {1}{K}, \tag {19}
$$
which is the density of discrete uniform distribution. Therefore, the  $\mathcal{H}(X)$  gets the great value  $\log K$  when the probability distribution conforms to a Discrete Uniform Distribution [14].
Then Jensen-Shannon Divergence (JSD) [16] is used to measure the distance between the probability distribution  $P \sim p(x)$  and the Discrete Uniform Distribution  $Q \sim q(x)$ .
$$
\begin{array}{l} \mathcal {J} (P \parallel Q) = \frac {1}{2} \sum_ {i = 1} ^ {K} p (x _ {i}) \log (\frac {p (x _ {i})}{p (x _ {i}) + q (x _ {i})}) + \frac {1}{2} \sum_ {i = 1} ^ {K} q (x _ {i}) \log (\frac {q (x _ {i})}{p (x _ {i}) + q (x _ {i})}) + 1 \\ = \frac {1}{2} \left(\sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log p \left(x _ {i}\right) - \sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log \left(p \left(x _ {i}\right) + q \left(x _ {i}\right)\right) \right. \tag {20} \\ + \sum_ {i = 1} ^ {K} q \left(x _ {i}\right) \log q \left(x _ {i}\right) - \sum_ {i = 1} ^ {K} q \left(x _ {i}\right) \log \left(p \left(x _ {i}\right) + q \left(x _ {i}\right)\right)) + 1. \\ \end{array}
$$
Since  $q(x)$  conforms to the Discrete Uniform Distribution,  $q(x_{i}) = \frac{1}{K}$  and  $\sum_{i=1}^{K} q(x_{i}) \log q(x_{i}) = -\log K$ . The JSD function (Eq. 20) can be converted to:
$$
\begin{array}{l} \mathcal {J} (P \parallel Q) = \frac {1}{2} \left(\sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log p \left(x _ {i}\right) - \sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log \left(\frac {1}{K} + p \left(x _ {i}\right)\right) \right. \\ - \log K - \frac {1}{K} \sum_ {i = 1} ^ {K} \log \left(\frac {1}{K} + p \left(x _ {i}\right)\right) + 1 \tag {21} \\ = \frac {1}{2} \left(\sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log p \left(x _ {i}\right) - \sum_ {i = 1} ^ {K} \left(\frac {1}{K} + p \left(x _ {i}\right)\right) \log \left(\frac {1}{K} + p \left(x _ {i}\right)\right) - \log K\right) + 1. \\ \end{array}
$$
Note that there are two components affecting the  $\mathcal{J}(P\parallel Q),\sum_{i = 1}^{K}p(x_{i})\log p(x_{i})$  and  $\sum_{i = 1}^{K}\left(\frac{1}{K} +p(x_i)\right)\log \left(\frac{1}{K} +p(x_i)\right)$ . For convenience, let:
$$
A (P) = - \mathcal {H} _ {P \sim p (x)} (X) = \sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log p \left(x _ {i}\right), \tag {22}
$$
$$
B (P) = \sum_ {i = 1} ^ {K} \left(\frac {1}{K} + p \left(x _ {i}\right)\right) \log \left(\frac {1}{K} + p \left(x _ {i}\right)\right).
$$
It can be seen that  $\mathcal{J}(P\parallel Q)\propto (A(P) - B(P))$  . When  $P\sim p(x)$  is closer to the Discrete Uniform Distribution  $Q\sim q(x)$  , i.e.,  $\mathcal{J}(P\parallel Q)\downarrow$  , three cases are possible for  $A(P)$  and  $B(P)$  ..
a.  $A(P)\downarrow$  and  $B(P)\downarrow$  , but the former has a larger reduction, i.e.,  $\Delta A(P) - \Delta B(P) <   0$
b.  $A(P)\downarrow$  and  $B(P)\uparrow$
c.  $A(P)\uparrow$  and  $B(P)\uparrow$ , but the latter has a larger growth, i.e.,  $\Delta A(P) - \Delta B(P) < 0$ .
$\Delta \to 0$  is the amount of function change.  $A(P)$  are both declining in cases of  $a$  and  $b$ , reflecting the increase of  $H_{Q \sim q(x)}(X)$  according to Eq. 22. Thus, only case  $c$  possibly induces the decrease of  $H_{Q \sim q(x)}(X)$  with the decrease of  $\mathcal{J}(P \parallel Q)$ . If there is no case suit  $c$ , the theorem is proved.
Let  $f(x) = x \log x$ . Its first-order derivative and second-order derivative are  $f(x)' = \log x + 1$  and  $f(x)'' = \frac{1}{x} > 0$ , respectively. It is a convex function obtaining the minimal value at  $x = \frac{1}{e}$ .
Due to  $\sum_{i=1}^{K} p(x_i) = 1$ , when  $p(x_a) \to p(x_a) + \Delta$ , there must be a  $p(x_b) \to p(x_b) - \Delta$ . Subject to  $A(P) \uparrow$ , Eq. 23 should be satisfied:
$$
\Delta A (P) = f \left(p \left(x _ {a}\right)\right) ^ {\prime} - f \left(p \left(x _ {b}\right)\right) ^ {\prime} = \log \frac {p \left(x _ {a}\right)}{p \left(x _ {b}\right)} > 0 \Longrightarrow p \left(x _ {a}\right) > p \left(x _ {b}\right). \tag {23}
$$
When  $0 < p(x_{b}) - \Delta < p(x_{b}) < p(x_{a}) < p(x_{a}) + \Delta < 1$ , we have:
$$
\begin{array}{l} \Delta A (P) - \Delta B (P) = \left(f (p (x _ {a})) ^ {\prime} - f (p (x _ {b})) ^ {\prime}\right) - \left(f (p (x _ {a}) + \frac {1}{K}) ^ {\prime} - f (p (x _ {b}) + \frac {1}{K}) ^ {\prime}\right) \\ = \log \frac {p \left(x _ {a}\right)}{p \left(x _ {a}\right) + \frac {1}{K}} - \log \frac {p \left(x _ {b}\right)}{p \left(x _ {b}\right) + \frac {1}{K}} \tag {24} \\ = \log (1 + \frac {1}{K p \left(x _ {b}\right)}) - \log (1 + \frac {1}{K p \left(x _ {a}\right)}) > 0. \\ \end{array}
$$
It indicates that case  $c$  is an impossible case when  $\mathcal{J}(P\parallel Q)\downarrow$ . Thus, only  $a$  and  $b$  are possible when  $\mathcal{J}(P\parallel Q)\downarrow$ . In these two cases,
$$
\mathcal {J} (P \| Q) \downarrow \longrightarrow A (P) \downarrow \longrightarrow H _ {P \sim p (x)} (X) \uparrow . \tag {25}
$$
$$
\mathcal {J} (P \| Q) \propto \frac {1}{\mathcal {H} (X)}, \tag {26}
$$
demonstrating that the closer to a uniform distribution, the larger the information entropy is. It concludes the proof.
F The Detailed Video Visualization
In Figure 6 of the main paper, we present several video fragments with their uncertainty. In Figure 8, more detailed videos are displayed to illustrate the uncertainty degree of the videos.
Figure 8(a) is a video with a 0.881 uncertainty score. Actually, it is difficult to understand the meaning expressed in the video. Obviously, as the uncertainty diminishes, the number of scenes in the videos decreases, and the videos become easier to understand.
Figure 8: The detailed video visualization of data uncertainty on MSR-VTT.