SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the other hand, consider the measurable selection  $T: \mathcal{X} \to \mathcal{Y}$  for the set-valued map  $x \mapsto \arg \min_{y \in \mathcal{Y}} (c(x, y) - f(y))$ . It exists thanks to [2, Theorem 18.19]. As a result, for the deterministic plan  $\pi^T = [\mathrm{id}, T] \sharp \mathbb{P}$ , the minimum in (31) is indeed attained. Therefore, (31) is the equality. We combine (30) and (31) and obtain
$$
\operatorname {C o s t} _ {w} (\mathbb {P}, \mathbb {Q}) = \min  _ {\pi \in \Pi^ {w} (\mathbb {P}, \mathbb {Q})} \int_ {\mathcal {X} \times \mathcal {Y}} c (x, y) d \pi (x, y) = \sup  _ {f \leq 0} \left\{\int_ {\mathcal {X}} f ^ {c} (x) d \mathbb {P} (x) + w \int_ {\mathcal {Y}} f (y) d \mathbb {Q} (y) \right\}. \tag {32}
$$
It remains to prove that sup in the right part is actually attained at some non-positive  $f^{*} \in \mathcal{C}(\mathcal{V})$ . Let  $f_{1}, f_{2}, \dots \in \mathcal{C}(\mathcal{V})$  be a sequence of non-positive functions for which  $\lim_{n \to \infty} \left\{ \int_{\mathcal{X}} f_{n}^{c}(x) d\mathbb{P}(x) + w \int_{\mathcal{Y}} f_{n}(y) d\mathbb{Q}(y) \right\} = \operatorname{Cost}_{w}(\mathbb{P}, \mathbb{Q})$ . For  $g \in \mathcal{C}(\mathcal{X})$ , we define the  $(c, -)$ -transform  $g^{(c, -)}(y) \stackrel{\text{def}}{=} \min \left[ \min_{x \in \mathcal{X}} \left( c(x, y) - g(x) \right), 0 \right] \leq 0$ . It yields a (uniformly) continuous non-positive function satisfying  $|g^{(c, -)}(y) - g^{(c, -)}(y')| \leq \omega \left( \| y - y' \|_{\mathcal{Y}} \right)$ , where  $\omega$  is the modulus of continuity of  $c(x, y)$ . This statement can be derived analogously to the proof of Proposition 1.
Before going further, let us highlight two important facts which we are going to use below. Consider any  $g \in \mathcal{C}(\mathcal{X})$  and  $0 \geq h \in \mathcal{C}(\mathcal{Y})$  satisfying  $g(x) + h(y) \leq c(x,y)$  for all  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ . First, from the definition of  $(c, - )$ -transform, one can see that for all  $(x,y) \in \mathcal{X} \times \mathcal{Y}$  it holds that  $0 \geq g^{(c, - )} \geq h$  and
$$
g (x) + h (y) \leq g (x) + g ^ {(c, -)} (y) \leq c (x, y), \tag {33}
$$
i.e.,  $(g, g^{(c, -)})$  also satisfies the assumptions of  $(g, h)$ . Second, from the definition of  $c$ -transform, it holds that  $h^c \geq g$  and
$$
g (x) + h (y) \leq h ^ {c} (x) + h (y) \leq c (x, y), \tag {34}
$$
i.e., the pair  $(h^c, h)$  satisfies the same assumptions as  $(g, h)$ .
Now we get back to our sequence  $f_{1}, f_{2}, \ldots$ . For each  $n$  and  $(x, y) \in \mathcal{X} \times \mathcal{Y}$ , we have  $f_{n}^{c}(x) + f_{n}(y) \leq c(x, y)$ . Next,
$$
f _ {n} ^ {c} (x) + f _ {n} (y) \leq f _ {n} ^ {c} (x) + \left(f _ {n} ^ {c}\right) ^ {(c, -)} (y) \leq \left(\left(f _ {n} ^ {c}\right) ^ {(c, -)}\right) ^ {c} (x) + \left(f _ {n} ^ {c}\right) ^ {(c, -)} (y) \quad [ \leq c (x, y) ], \tag {35}
$$
where we first used (33) with  $(g,h) = (f_n^c,f_n)$  and then used (34) with  $(g,h) = (f_n^c,(f_n^c)^{(c, - )})$ . In particular, we have  $f_{n}^{c}\leq \left((f_{n}^{c})^{(c, - )}\right)^{c}$  and  $f_{n}\leq (f_{n}^{c})^{(c, - )}$ . We sum these inequalities with weights 1 and  $w$ , and for all  $(x,y)\in \mathcal{X}\times \mathcal{Y}$  obtain
$$
f _ {n} ^ {c} (x) + w f _ {n} (y) \leq \left(\left(f _ {n} ^ {c}\right) ^ {(c, -)}\right) ^ {c} (x) + w \left(f _ {n} ^ {c}\right) ^ {(c, -)} (y) = h _ {n} ^ {c} (x) + w h _ {n} (y), \tag {36}
$$
where for convenience we denote  $h_n \stackrel{\text{def}}{=} (f_n^c)^{(c, -)}$ . Integrating (36) with  $(x, y) \sim \mathbb{P} \times \mathbb{Q}$  yields
$$
\int_ {\mathcal {X}} f _ {n} ^ {c} (x) d \mathbb {P} (x) + w \int_ {\mathcal {Y}} f _ {n} (y) d \mathbb {Q} (y) \leq \int_ {\mathcal {X}} h _ {n} ^ {c} (x) d \mathbb {P} (x) + w \int_ {\mathcal {Y}} h _ {n} (y) d \mathbb {Q} (y). \tag {37}
$$
This means that potential  $h_n$  provides not smaller dual objective value than  $f_n$ . As a result, sequence  $h_1, h_2 \ldots$  also satisfies  $\lim_{n \to \infty} \left\{ \int_{\mathcal{X}} h_n^c(x) d\mathbb{P}(x) + w \int_{\mathcal{Y}} h_n(y) d\mathbb{Q}(y) \right\} = \operatorname{Cost}_w(\mathbb{P}, \mathbb{Q})$ . Now we forget about  $f_1, f_2, \ldots$  and work with  $h_1, h_2, \ldots$ .
All the functions  $h_n$  are uniformly equicontinuous as they share the same modulus of continuity  $\omega$  because they are  $(c, -)$ -transforms by their definition. Let  $v_n(y) \stackrel{\text{def}}{=} h_n(y) - \max_{y' \in \mathcal{Y}} h_n(y')$ . This function is also non-positive and uniformly continuous as well. Note that  $v_n$  provides the same dual objective value as  $h_n$ . This follows from the definition of  $v^c = h^c + \max_{y' \in \mathcal{Y}} h_n(y')$ . Here the additive constant vanishes, i.e.,  $v_n^c(x) + v_n(y) = h_n^c(x) + h_n(y)$ . At the same time,  $v_n$  are all uniformly bounded. Indeed, let  $y_n \in \mathcal{Y}$  be any point where  $v_n(y_n) = 0$ . Then for all  $y \in \mathcal{Y}$  it holds that  $|v_n(y)| = |v_n(y) - v_n(y_n)| \leq \omega(\|y - y_n\|_Y) \leq \omega(\mathrm{diam}(\mathcal{Y}))$ . Therefore, by the Arzelà-Ascoli theorem, there exists a subsequence  $v_{n_k}$  uniformly converging to some  $f^* \in \mathcal{C}(X)$ . As all  $v_{n_k} \leq 0$ , it holds that  $f^* \leq 0$  as well. It remains to check that  $f^*$  attains the supremum in (32).
To begin with, we prove that  $v_{n_k}^c$  uniformly converges to  $(f^{*})^{c}$ . Denote  $\| v_{n_k} - f^*\|_{\infty} = \epsilon_k$ . For all  $(x,y)\in \mathcal{X}\times \mathcal{Y}$ , we have
$$
c (x, y) - f ^ {*} (y) - \epsilon_ {k} \leq c (x, y) - v _ {n _ {k}} (y) \leq c (x, y) - f ^ {*} (y) + \epsilon_ {k} \tag {38}
$$
since  $|v_{n_k}(y) - f^* (y)| \leq \| v_{n_k} - f^*\|_\infty < \epsilon$ . We take  $\min_{y\in \mathcal{V}}$  in (38) and obtain  $(f^{*})^{c}(x) - \epsilon_{k}\leq v_{n_{k}}^{c}(x)\leq (f^{*})^{c}(x) + \epsilon_{k}$ . As this holds for all  $x\in \mathcal{X}$ , we have just proved that  $\| v_{n_k}^c -(f^*)^c\|_\infty < \epsilon_k$ . This means that  $v_{n_k}^c$  uniformly converges to  $(f^{*})^{c}$  as well since  $\lim_{k\to \infty}\| v_{n_k} - f^*\|_\infty = \lim_{k\to \infty}\epsilon_k = 0$ . Thanks to the uniform convergence, we have
$$
\begin{array}{l} \operatorname {C o s t} _ {w} (\mathbb {P}, \mathbb {Q}) = \lim  _ {k \rightarrow \infty} \left\{\int_ {\mathcal {X}} \left(v _ {n _ {k}}\right) ^ {c} (x) d \mathbb {P} (x) + \int_ {\mathcal {Y}} v _ {n _ {k}} (y) d \mathbb {Q} (y) \right\} = \\ \int_ {\mathcal {X}} \left(f ^ {*}\right) ^ {c} (x) d \mathbb {P} (x) + \int_ {\mathcal {Y}} f ^ {*} (y) d \mathbb {Q} (y). \tag {39} \\ \end{array}
$$
We conclude that  $f^{*}$  is a maximizer of (32) that we seek for.
Bibliographical remark. There exists a duality formula for partial OT (4), see [10, §2] which can be reduced to duality formula to IT problem (12). However, it is hard to relate the resulting formula with ours (13). We do not know how to derive one formula from the other. More importantly, it is unclear how to turn their formula to the computational algorithm. Our formula provides an opportunity to do this by using the saddle point reformulation of the dual problem which nowadays becomes standard for neural OT, see [43, 20, 61]. We will give further comments after the next proof. The second part of the derivation of our formula (existence of a maximizer  $f^{*}$ ) is inspired by the [62, Proposition 1.11] which shows the existence of maximizers for standard OT (2).
Proof of Theorem 5. By the definition of  $f^{*}$ , we have
$$
\begin{array}{l} \operatorname {C o s t} _ {w} (\mathbb {P}, \mathbb {Q}) = \\ \min  _ {T: \mathcal {X} \rightarrow \mathcal {Y}} \mathcal {L} (f ^ {*}, T) = \min  _ {T: \mathcal {X} \rightarrow \mathcal {Y}} \int_ {\mathcal {X}} \left\{c (x, T (x)) - f ^ {*} (T (x)) \right\} d \mathbb {P} (x) + w \int_ {\mathcal {Y}} f ^ {*} (y) d \mathbb {Q} (y) \leq (40) \\ \int_ {\mathcal {X}} \left\{c (x, T ^ {*} (x)) - f ^ {*} (T ^ {*} (x)) \right\} d \mathbb {P} (x) + w \int_ {\mathcal {Y}} f ^ {*} (y) d \mathbb {Q} (y) = (41) \\ \int_ {\mathcal {X}} c (x, T ^ {*} (x)) d \mathbb {P} (x) - \int_ {\mathcal {X}} f ^ {*} (y) d [ T ^ {*} \sharp \mathbb {P} ] (y) + w \int_ {\mathcal {Y}} f ^ {*} (y) d \mathbb {Q} (y) = \\ \end{array}
$$
$$
\operatorname {C o s t} _ {w} (\mathbb {P}, \mathbb {Q}) + \underbrace {\int_ {y} f ^ {*} (y) d \left[ w \mathbb {Q} - T ^ {*} \sharp \mathbb {P} \right] (y)} _ {\leq 0 (\text {L e m m a} 1)} \leq \operatorname {C o s t} _ {w} (\mathbb {P}, \mathbb {Q}). \tag {42}
$$
This means that all the inequalities in (40)-(42) are equalities. Since (40) equals (41), we have  $T^{*} \in \operatorname*{arg  min}_{T: \mathcal{X} \to \mathcal{Y}} \mathcal{L}(f^{*}, T)$ .
Bibliographic remark (theoretical part). The idea of the theorem is similar to that of [61, Lemma 4.2], [24, Lemma 3], [43, Lemma 4], [20, Theorem 2] which prove that their respective saddle point objectives  $\max_f\min_T\mathcal{L}(f,T)$  can be used to recover optimal  $T^{*}$  from some optimal saddle points  $(f^{*},T^{*})$ . Our functional (15) differs, and we have the constraint  $f\leq 0$ . We again emphasize here that not for all the saddle points  $(f^{*},T^{*})$  it necessarily holds that  $T^{*}$  is the IT map, see the discussion in limitations (Appendix A).
Bibliographic remark (algorithmic part). To derive our saddle point optimization problem (15), we use the  $c$ -transform expansion proposed by [53] in the context of Wasserstein GANs and later explored by [40, 43, 61, 20, 24, 31] in the context of learning OT maps. That is, our resulting algorithm 1 overlaps with the standard maximin neural OT solver, see, e.g., [24, Algorithm 1]. The difference is in the constraint  $f \leq 0$  and the additional multiplier  $w \geq 1$ .
Proof of Proposition 4. From the proof of Theorem 5, we see that  $\int_{\mathcal{V}}f^{*}(y)d\big[w\mathbb{Q} - T^{*}\sharp \mathbb{P}\big](y) = 0$  Recall that  $f^{*}\leq 0$  . This means that  $f(y) = 0$  for  $y\in \operatorname {Supp}\bigl (w\mathbb{Q} - T^{*}\sharp \mathbb{P}\bigr)$  . Indeed, assume the opposite, i.e., there exists some  $y\in \operatorname {Supp}\bigl (w\mathbb{Q} - T^{*}\sharp \mathbb{P}\bigr)$  for which  $f(y) <   0$  . In this case, the same holds for all  $y^\prime$  in a small neighborhood  $U$  of  $y$  as  $f$  is continuous. At the same time,  $\int_{\mathcal{V}}f^{*}(y)d\big[w\mathbb{Q} - T^{*}\sharp \mathbb{P}\big](y)\leq \int_{U}f^{*}(y)d\big[w\mathbb{Q} - T^{*}\sharp \mathbb{P}\big](y) <   0$  since  $\left[w\mathbb{Q} - T^{*}\sharp \mathbb{P}\right]$  is a non-negative measure satisfying  $\left[w\mathbb{Q} - T^{*}\sharp \mathbb{P}\right](U) > 0$  by the definition of the support. This is a contradiction. To finish the proof it remains to note that  $\mathrm{Supp}(\mathbb{Q})\setminus \mathrm{Supp}(T^{*}\sharp \mathbb{P})\subset \mathrm{Supp}(w\mathbb{Q} - T^{*}\sharp \mathbb{P})$  , i.e.,  $f(y) = 0$  for  $y\in \operatorname {Supp}(\mathbb{Q})\setminus \operatorname {Supp}(T^{*}\sharp \mathbb{P})$  as well.
Bibliographical remark. Treating functional  $\mathcal{L}(f,T)$  in (16) as a Lagrangian, Proposition 4 can be viewed as a consequence of the complementary slackness in the Karush-Kuhn-Tucker conditions [37].
G Additional Experimental Illustrations
G.1 Comparison with the Closed-form Solution for ET
In this section, we conduct a Swiss2Ball experiment in 2D demonstrating that for the sufficiently large parameter  $w$ , IT maps become fine approximations of the ground-truth ET map. We define source measure  $\mathbb{P}$  as a uniform distribution on a swiss roll centered at  $(0,0)$ . Target measure  $\mathbb{Q}$  is a uniform distribution on a ball centered at  $(0,0)$  with radius  $R = 0.5$ , i.e.,  $\operatorname{Supp}(\mathbb{Q}) = B(0,0.5)$ . We note that the supports of source and target measures are partially overlapping. In the proposed setup, the solution to ET problem (5) has a closed form:  $T(x) = x \cdot 1_{x \in B((0,0),0.5)} + x \cdot \frac{R}{\|x\|_2} \cdot 1_{x \notin B((0,0),0.5)}$ , see Fig. 20f. We provide the learned IT maps for  $w \in \{1,3/2,2,32\}$ , see Fig. 20b-20e. The qualitative and quantitative results show that with the increase of  $w$  our IT maps become closer and closer to the ground-truth ET one, see Table 8.
(b) IT map,  $w = 1$
(c) IT map,  $w = \frac{3}{2}$ .
Figure 20: Incomplete Transport (IT) maps learned with  $c(x, y) = \| x - y\|_2^2$  transport cost and ground-truth Extremal Transport (ET) map in 'Swiss2Ball' experiment.
(d) IT map,  $w = 2$
(e) IT map,  $w = 32$
(f) Extremal transport map.
w=1	w=3/2	w=2	w=32
MSE(ˆT,T*)	0.0136	0.0026	0.0009	7.98e-06
G.2 Solving Fake Solutions Issue with Weak Kernel Cost
As we discuss in Appendix A, saddle-point neural OT methods (including our IT algorithm) may suffer from fake solutions issue. As it is proved in [42], this issue can be eliminated by considering OT with the so-called weak kernel cost functions. In this section, we test the effect of using them in our IT algorithm. Specifically, we demonstrate the example where IT algorithm with the cost function  $c(x,y) = \| x - y\| _2$  struggles from fake solutions, while IT with the same cost endowed with weak kernel regularization, i.e., kernel cost [42, Equation (16)] with parameters  $\alpha = 1$ ,  $\gamma = 0.4$ , resolves the issue. Following [42], we consider stochastic version of IT map  $T(x,z)$  using noise  $z\sim \mathcal{N}(0,\mathcal{I})$  as an additional input.
We design the Ball2Circle example in 2D, where input  $\mathbb{P}$  is a uniform distribution on a ball and target  $\mathbb{Q}$  - on a ring embracing the ball. The solution of ET problem is an internal circuit of a ring, see Fig. 21a. We learn IT maps for  $c(x,y) = \| x - y\| _2$  , with  $(\gamma = 0.4)$  or without  $(\gamma = 0)$  kernel regularization for weights  $w\in \{1,2,32\}$
(b) IT map,  $w = 1$  (iter=7K)
(c) IT map,  $w = \frac{3}{2}$  (iter=8K)
(d) IT map,  $w = 2$  (iter=7K)
(e) IT map,  $w = 2$  (iter=8K).
(f) IT map,  $w = 32$
(a) Input, target measures and ET map.
Figure 21: Incomplete Transport (IT) maps learned with  $c(x,y) = \| x - y\| _2$  transport cost in 'Ball2Circle' experiment. We observe that training is highly unstable for  $w\in \{1,2\}$ , see the solutions for nearby iterations of training - (b-c) and (d-e) respectively. Increase of the weight  $w$  helps to improve the stability (f).
(c) IT map,  $w = 2$
d) IT map,  $w = 32$
Figure 22: Incomplete transport (IT) maps learned with weak kernel cost in 'Ball2Circle' experiment for a fixed noise  $z$ . Usage of the kernel regularization + stochastic map  $T(x,z)$  helps to overcome instability issues for weights  $w \in \{1,2\}$ .
Table 8: MSE(  $\hat{T},T^{*}$  ) between learned IT maps  $\hat{T}$ $(w\in \{1,3 / 2,2,32\})$  and ground-truth ET map  $T^{*}$
method / weight	w = 1	w = 2	w = 32
w/o kernel regularization	-	-	0.0022
with kernel regularization	0.1377	0.0730	0.0495
Table 9:  $\mathrm{MSE}(\hat{T},T^{*})$  between IT maps  $(w\in \{1,2,32\})$  learned with  $c(x,y) = \| x - y\| _2$  (with and without weak kernel regularization) and the ground-truth ET map. MSE for IT map learned with kernel regularization is larger (see  $w = 32$ ) than without the regularization since it introduces small bias to the optimization.
Discussion. We observe that without kernel regularization training of IT method is highly unstable (for  $w \in \{1,2\}$ ), see Fig. 21b-21d. Interestingly, with the increase of the weight  $w$ , the issue disappears and for  $w = 32$ , IT map is close to the ground-truth ET one (Fig. 21f).
The usage of kernel regularization helps to improve stability of the method, see Fig. 22. However, while MSE between learned and ground-truth ET map drops with the increase of  $w$ , for  $w = 32$  it is bigger than that of learned IT maps without regularization. It is expected, since using regularizations usually leads to some bias in the solutions.
Thus, the example shows that (a) fake solutions may be a problem, (b) kernel regularization from [42] may help to deal with them. However, further studying this aspect is out of the scope of the paper.
G.3 Perceptual Cost
In this section, we show that the stated conclusions hold true for the transport costs other than  $\ell^2$ . For this purpose, we perform additional experiments using perceptual transport cost from [24]. We use the same hyperparameters as in our experiments with  $\ell^2$  cost.
Figure 23: Celeba (female)  $\rightarrow$  anime ( $64 \times 64$  image size, perceptual cost).
Metrics	w = 1	w = 2	w = 4	w = 8
Test FID	9.21	12.98	17.24	22.08
Test perceptual cost	0.954	0.794	0.667	0.545
Test ℓ² cost	0.303	0.209	0.153	0.103
Table 10: Test FID and  $\ell^2$ , perceptual transport costs of our IT maps (learned with perceptual transport cost).
Experimental results. Qualitative results show that that similarity of input images  $x$  and the images  $\hat{T}(x)$  translated by our IT method trained with perceptual cost grows with the increase of the parameter  $w$ , see Figure 23. In 10, we quantitatively verify these observations by showing that both perceptual and  $\ell^2$  mean transport costs between input and translated images decrease with the increase of  $w$ . Interestingly, we see that IT trained with perceptual cost yields smaller FID than IT method trained with  $\ell^2$  cost.
G.4 Bigger Weight Parameters
In this section, we present the results of IT method trained with  $\ell^2$  cost and parameters  $w = 16,32$ . We see that in contrast to CycleGAN which does not translate face images to anime images in case of big parameters  $\lambda$ , our IT method continues to translate faces to anime even for big parameters  $w$ . We quantify the obtained results in Table 11. As expected, mean transport costs are decreasing with the increase of  $w$ , while FID is slightly increasing. We present the qualitative results for additional weights in Figure 24.
Metrics	Main results	Additional results
w = 1	w = 2	w = 4	w = 8	w = 16	w = 32
Test FID	14.65	20.79	22.18	22.84	24.86	28.28
Test ℓ2 cost	0.297	0.154	0.133	0.094	0.091	0.083
Table 11: Test FID and  ${\ell }^{2}$  transport costs of our IT maps.
G.5 Additional Results for  $\ell^2$  Cost
Figure 24: Celeba (female)  $\rightarrow$  anime  $(64\times 64)$
Figure 25: Handbag  $\rightarrow$  shoes (128×128).
Figure 26: Textures  $\rightarrow$  chairs  $(64\times 64)$
Figure 27: Ffhq  $\rightarrow$  comics (128×128).