SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mu$  and  $\sigma$  represent the mean and covariance value for  $\mathrm{Conv}_i(\pmb{x})$ . Notably,  $\mu$  and  $\sigma$  are not learnable parameters. Moreover, the parameters  $\mu_i$  and  $\sigma_i$  has the same dimension as the output of  $\mathrm{Conv}_i(\pmb{x})$ . After that, we use a transposed convolution layer TranConv to turn the feature maps generated by the above operations into a real instance, which has the same dimension as  $\pmb{x}$ .
Putting all above, we generate the hard-generalized domain samples  $\hat{x}$  following:
$$
\hat {\boldsymbol {x}} = \frac {1}{\sum w _ {i}} \sum_ {i} w _ {i} \cdot \operatorname {t a h n} \left(\operatorname {T r a n C o n v} \left(\sigma_ {i} \cdot \frac {\operatorname {C o n v} _ {i} (\boldsymbol {x}) - \mu}{\sigma} + \mu_ {i}\right)\right), \tag {10}
$$
where tahn represents the tahn activation function.  $w_{i}$  is a scalar and weights the contribution of each activated instance produced by TransposedConv to  $\hat{\pmb{x}}$ .  $w_{i}$  is randomly sampled from normal distribution  $w_{i} \sim N(0,1)$ . Notably, for each input  $\pmb{x}$ , we first up-sample it to  $224 \times 224$  size and down-sample produced  $\hat{\pmb{x}}$  to the original size for  $\pmb{x}$ .
A.2 The Optimization Process
During the optimization process of Eq. (2), we first initialized a surrogate model  $f(\cdot; \boldsymbol{w})$  and a benign dataset  $\mathcal{D}$ . Then during each iteration for solving the bi-level optimization Eq. (2), we first minimize the  $I(z; \hat{z})$  and  $\mathcal{L}_c$  by optimizing the parameters of our proposed transformation module:
$$
\min  _ {\boldsymbol {\theta}} \mathbb {E} _ {p (\boldsymbol {z}, \hat {\boldsymbol {z}})} \left[ I \left(\boldsymbol {z} \left(\boldsymbol {w} ^ {*}\right); \hat {\boldsymbol {z}} \left(\boldsymbol {\theta}, \boldsymbol {w} ^ {*}\right)\right) + \lambda_ {1} \mathcal {L} _ {c} \left(\boldsymbol {z} \left(\boldsymbol {w} ^ {*}\right), \hat {\boldsymbol {z}} \left(\boldsymbol {\theta}, \boldsymbol {w} ^ {*}\right)\right) \right]. \tag {11}
$$
Table 3: The configuration for each convolution layer.
Layer	Kernel Size	Input Channel	Output Channel
Conv1	5x5	3	3
Conv2	9x9	3	3
Conv3	13x13	3	3
Conv4	17x17	3	3
After that, we maximize  $I(z; \hat{z})$  and minimize the training loss by optimizing the parameters  $w$ :
$$
\min  _ {\boldsymbol {w}} \left[ \mathbb {E} _ {(\boldsymbol {x}, y) \sim \mathcal {D}} \left[ \mathcal {L} \left(f \left(G _ {d} (\boldsymbol {x}; \boldsymbol {\theta}); \boldsymbol {w}\right), y\right) + \mathcal {L} \left(f (\boldsymbol {x}; \boldsymbol {w}), y\right) \right] - \lambda_ {2} \mathbb {E} _ {p (\boldsymbol {z}, \hat {\boldsymbol {z}})} \left[ I (\boldsymbol {z} (\boldsymbol {w}); \hat {\boldsymbol {z}}) \right]. \right. \tag {12}
$$
Since  $I(z; \hat{z})$  is intractable, we propose to optimize its upper bound instead:
$$
I (\boldsymbol {z}; \hat {\boldsymbol {z}}) = \mathbb {E} _ {p (\boldsymbol {z}, \hat {\boldsymbol {z}})} \left[ \log \frac {p (\hat {\boldsymbol {z}} | \boldsymbol {z})}{p (\hat {\boldsymbol {z}})} \right] \leq \mathbb {E} _ {p (\boldsymbol {z}, \hat {\boldsymbol {z}})} [ \log p (\hat {\boldsymbol {z}} | \boldsymbol {z}) ] - \mathbb {E} _ {p (\boldsymbol {z}) p (\hat {\boldsymbol {z}})} [ \log p (\hat {\boldsymbol {z}} | \boldsymbol {z}) ]. \tag {13}
$$
Since the conditional distribution  $p(\hat{z} | z)$  is also intractable thus the upper bound of  $I(z; \hat{z})$  can't be optimized, we follow previous work to adopt a variational distribution  $q(\hat{z} | z)$  to approximate the upper bound of  $I(z; \hat{z})$ :
$$
I (\boldsymbol {z}; \hat {\boldsymbol {z}}) \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \log q \left(\hat {\boldsymbol {z}} _ {i} \mid \boldsymbol {z} _ {i}\right) - \frac {1}{N} \sum_ {j = 1} ^ {N} \log q \left(\hat {\boldsymbol {z}} _ {j} \mid \boldsymbol {z} _ {i}\right) \right], \tag {14}
$$
where  $q(\hat{z} | z)$  is obtained by employing the backbone neural network to approximate.
We optimize the above bi-level optimization Eq. (2) with 100 iterations. We set the learning rate as 0.005 for optimizing the parameters of the proposed transformation module and 0.001 for parameters for the backbone model  $f(\cdot)$  following [50]. The batch size is 64. For both the transformation module and the backbone model  $f(\cdot)$ , we use SGD [54] as the optimizer with Nesterov momentum and weight decay rate of 0.0005. We use ResNet-18 as the backbone model for extracting  $z$  and  $\hat{z}$  throughout the paper. We introduce  $\lambda_{1}$  and  $\lambda_{2}$  for balancing each optimization objective. Following the implementation of [50], we set  $\lambda_{1}$  and  $\lambda_{2}$  as 0.1 and 1.0.
B The Proof for Theorem 1
$$
\begin{array}{l} \mathcal {R} _ {\mathcal {T}} (f) \leq \frac {c}{2 \left(1 - e ^ {- c}\right)} \widehat {\mathcal {R}} _ {T} (f) + \frac {c}{1 - e ^ {- c}} \beta_ {\infty} (\mathcal {T} \| \mathcal {S}) \widehat {\mathcal {R}} _ {S} (f) + \Omega \\ + \frac {1}{1 - e ^ {- c}} \left(\frac {1}{n _ {t}} + \frac {\beta_ {\infty} (\mathcal {T} \| \mathcal {S})}{n _ {s}}\right) \left(2 \mathrm {K L} (f \| \pi) + \ln \frac {2}{\xi}\right), \tag {15} \\ \end{array}
$$
$$
\beta_ {\infty} (\mathcal {T} \| \mathcal {S}) = \sup  _ {(\boldsymbol {x}, y) \in \operatorname {S U P P} (\mathcal {S})} \left(\frac {\mathcal {P} _ {(\boldsymbol {x} , y) \in \mathcal {T}}}{\mathcal {P} _ {(\boldsymbol {x} , y) \in \mathcal {S}}}\right) \geq 1, \tag {16}
$$
Proof. Theorem 6 in Germain et al.'s work [55] demonstrates that suppose in PAC Bayesian [40], for a target domain  $\mathcal{T}$  and a source domain  $S$ , any set of voters (candidate models)  $\mathcal{H}$ , any prior  $\pi$  over  $\mathcal{H}$  before any training, any  $\xi \in (0,1]$ , any  $c > 0$ , with a probability at least  $1 - \xi$  over the choices of  $S \sim S^{n_s}$  and  $T \sim T_{\mathcal{X}}^{n_t}$ , for the posterior  $f$  over  $\mathcal{H}$  after the joint training on  $S$  and  $T$ :
$$
\begin{array}{l} \mathcal {R} _ {\mathcal {T}} (f) \leq \frac {c}{2 (1 - e ^ {- c})} \widehat {\mathrm {d}} _ {T} (f) + \frac {c}{1 - e ^ {- c}} \beta_ {\infty} (\mathcal {T} \| \mathcal {S}) \widehat {\mathrm {e}} _ {S} (f) + \Omega \\ + \frac {1}{1 - e ^ {- c}} \left(\frac {1}{n _ {t}} + \frac {\beta_ {\infty} (\mathcal {T} \| \mathcal {S})}{n _ {s}}\right) \left(2 \mathrm {K L} (f \| \pi) + \ln \frac {2}{\xi}\right), \tag {17} \\ \end{array}
$$
where  $\mathcal{R}_{\mathcal{T}}(f)$  denotes the expected Gibbs risk of voter  $f$  over the target domain.  $\widehat{\mathrm{d}}_T(f)$  and  $\widehat{\mathrm{e}}_S(f)$  are the empirical estimation of the target voters' disagreement and the source joint error, measured over target and source datasets  $T$  and  $S$ , respectively.  $\Omega$  is a constant and  $\mathrm{KL}(\cdot)$  is the Kullback-Leibler divergence.  $\beta_{\infty}(\mathcal{T}\| S)$  measures the discrepancy between  $\mathcal{T}$  and  $S$ , defined as:
$$
\beta_ {\infty} (\mathcal {T} \| \mathcal {S}) = \sup  _ {(\boldsymbol {x}, y) \in \operatorname {S U P P} (\mathcal {S})} \left(\frac {\mathcal {P} _ {(\boldsymbol {x} , y) \in \mathcal {T}}}{\mathcal {P} _ {(\boldsymbol {x} , y) \in \mathcal {S}}}\right), \tag {18}
$$
where  $\mathrm{SUPP}(S)$  denotes the support of  $S$ .
In the following proof, in particular, the Gibbs risk  $\mathcal{R}_{\mathcal{A}}(f)$ , the voters' disagreement  $\mathrm{d}_A(f)$ , and the joint error  $\mathrm{e}_{\mathcal{A}}(f)$  of a certain domain  $\mathcal{A}$  are defined as follows.
$$
\mathcal {R} _ {\mathcal {A}} (f) = \underset {(\boldsymbol {x}, y) \sim \mathcal {A}} {\mathbb {E}} \underset {h \sim f} {\mathbb {E}} \mathbb {I} [ h (\boldsymbol {x}) \neq y ], \tag {19}
$$
$$
\mathrm {d} _ {\mathcal {A}} (f) = \underset {\boldsymbol {x} \sim \mathcal {A} _ {\mathcal {X}}} {\mathbb {E}} \underset {h \sim f} {\mathbb {E}} \underset {h ^ {\prime} \sim f} {\mathbb {E}} \mathbb {I} [ h (\boldsymbol {x}) \neq h ^ {\prime} (\boldsymbol {x}) ], \tag {20}
$$
$$
\mathrm {e} _ {\mathcal {A}} (f) = \underset {(\boldsymbol {x}, y) \sim \mathcal {A}} {\mathbb {E}} \underset {h \sim f} {\mathbb {E}} \underset {h ^ {\prime} \sim f} {\mathbb {E}} \underset {h ^ {\prime} \sim f} {\mathbb {I}} [ h (\boldsymbol {x}) \neq y ] \underset {h ^ {\prime}} {\mathbb {I}} [ h ^ {\prime} (\boldsymbol {x}) \neq y ], \tag {21}
$$
where  $\mathbb{I}[\mathrm{True}] = 1$  if the inner condition is true, and otherwise  $\mathbb{I}[\mathrm{False}] = 0$ , and  $\mathcal{A}_{\mathcal{X}}$  is the marginal distribution of domain  $\mathcal{A}$ .  $h$  and  $h'$  are votes sampled from the posterior distribution  $f$  over  $\mathcal{H}$ . With these definitions, studies [56, 57] reveal a relationship among the Gibbs risk, the voters' disagreement, and the joint error as
$$
\mathcal {R} _ {\mathcal {A}} (f) = \underset {(\boldsymbol {x}, y) \sim \mathcal {A}} {\mathbb {E}} \underset {h \sim f} {\mathbb {E}} \underset {h ^ {\prime} \sim f} {\mathbb {E}} \frac {\mathbb {I} [ h (\boldsymbol {x}) \neq h ^ {\prime} (\boldsymbol {x}) ] + 2 \mathbb {I} [ h (\boldsymbol {x}) \neq y \wedge h ^ {\prime} (\boldsymbol {x}) \neq y ]}{2} = \frac {1}{2} \mathrm {d} _ {\mathcal {A}} (f) + \mathrm {e} _ {\mathcal {A}} (f). \tag {22}
$$
In this case, we can extend this relationship to the empirical estimations (suppose a dataset  $A$  is sampled from domain  $\mathcal{A}$ ) as
$$
\widehat {\mathcal {R}} _ {A} (f) = \frac {1}{| A |} \sum_ {(\boldsymbol {x}, y) \sim_ {A}} \underset {h \sim f} {\mathbb {E}} \underset {h ^ {\prime} \sim f} {\mathbb {E}} \frac {\mathbb {I} [ h (\boldsymbol {x}) \neq h ^ {\prime} (\boldsymbol {x}) ] + 2 \mathbb {I} [ h (\boldsymbol {x}) \neq y \wedge h ^ {\prime} (\boldsymbol {x}) \neq y ]}{2} = \frac {1}{2} \widehat {\mathrm {d}} _ {A} (f) + \widehat {\mathrm {e}} _ {A} (f). \tag {23}
$$
Then we can use  $\widehat{\mathcal{R}}_T(f)$  and  $\widehat{\mathcal{R}}_S(f)$  to replace  $\widehat{\mathrm{d}}_T(f)$  and  $\widehat{\mathrm{e}}_S(f)$  in Eq. (17), respectively. In the end, we can follow Xu et al. [58] to regard these empirical risks as data quantity-irrelevant when analyzing the impact of data quantity.
Next, we focus on the proof of the numerical relationship  $\beta_{\infty}(\mathcal{T}\| \mathcal{S})\geq 1$ . First of all,  $\beta_{\infty}(\mathcal{T}\| \mathcal{S})$  comes from a more general definition that is parameterized by a real value  $q > 0$ , shown as
$$
\beta_ {q} (\mathcal {T} \| \mathcal {S}) = \left[ \underset {(\boldsymbol {x}, y) \sim \mathcal {S}} {\mathbb {E}} \left(\frac {\mathcal {P} _ {(\boldsymbol {x} , y) \in \mathcal {T}}}{\mathcal {P} _ {(\boldsymbol {x} , y) \in \mathcal {S}}}\right) ^ {q} \right] ^ {\frac {1}{q}}. \tag {24}
$$
For any  $q > 0$ ,  $\beta_q(\mathcal{T}||\mathcal{S})$  can be also written as a Renyi Divergence-based form [55], i.e.,
$$
\beta_ {q} (\mathcal {T} \| \mathcal {S}) = 2 ^ {\frac {q - 1}{q} D _ {q} (\mathcal {T} \| \mathcal {S})}, \tag {25}
$$
where  $D_q(\mathcal{T} \| \mathcal{S})$  is the Renyi Divergence between  $\mathcal{T}$  and  $\mathcal{S}$  with the order  $q$ . For Renyi Divergence with any order  $q > 0$ , there is a property of positivity [59], i.e.,  $D_q(\mathcal{T} \| \mathcal{S}) \geq 0$ . In this case, when  $q \to \infty$ , Eq. (25) becomes  $\beta_q(\mathcal{T} \| \mathcal{S}) = 2^{D_q(\mathcal{T} \| \mathcal{S})} \geq 1$ , and  $\beta_q(\mathcal{T} \| \mathcal{S}) = 2^{D_q(\mathcal{T} \| \mathcal{S})} = 1$  when  $\mathcal{T} = \mathcal{S}$ , in other words,  $D_q(\mathcal{T} \| \mathcal{S}) = 0$  when  $\mathcal{T} = \mathcal{S}$  [59].
C Technical Details for Generating Protected Dataset
C.1 The Optimization Solution for Generating Protected Dataset
Recall that Eq. (7) is formulated as follows.
$$
\left. \min  _ {\delta \subset \mathcal {B}} \left[ \mathbb {E} _ {(\hat {\boldsymbol {x}}, y) \sim \mathcal {T}} [ \mathcal {L} (f (\hat {\boldsymbol {x}}; \boldsymbol {w} (\delta)), y) ] - \lambda_ {3} \min  \left\{\mathbb {E} _ {(\overline {{\boldsymbol {x}}}, y) \sim \overline {{\mathcal {T}}} [ \mathcal {L} (f (\overline {{\boldsymbol {x}}}; \boldsymbol {w} (\delta)), y) ], \lambda_ {4} \right\} \right], \right. \tag {26}
$$
$$
s. t. \boldsymbol {w} (\boldsymbol {\delta}) = \arg \min _ {\boldsymbol {w}} \left[ \frac {1}{| \mathcal {D} _ {s} |} \sum_ {(\boldsymbol {x} _ {i}, y _ {i}) \in \mathcal {D} _ {s}} \mathcal {L} (f (\boldsymbol {x} _ {i} + \boldsymbol {\delta} _ {i}; \boldsymbol {w}), y _ {i}) + \frac {1}{| \mathcal {D} _ {b} |} \sum_ {(\boldsymbol {x} _ {j}, y _ {j}) \in \mathcal {D} _ {b}} \mathcal {L} (f (\boldsymbol {x} _ {j}; \boldsymbol {w}), y _ {j}) \right],
$$
The aforementioned problem is a standard bi-level problem, we following previous work [60, 61] to leverage gradient matching to solving it. Specifically, we first make the following definition:
$$
\mathcal {L} _ {t} = \mathbb {E} _ {(\hat {\boldsymbol {x}}, y) \sim \mathcal {T}} [ \mathcal {L} (f (\hat {\boldsymbol {x}}; \boldsymbol {w}), y) ] - \lambda_ {3} \min  \left\{\mathbb {E} _ {(\overline {{\boldsymbol {x}}}, y) \sim \overline {{\mathcal {T}}}} [ \mathcal {L} (f (\overline {{\boldsymbol {x}}} ; \boldsymbol {w}), y) ], \lambda_ {4} \right\}, \tag {27}
$$
$$
\mathcal {L} _ {i} = \frac {1}{| \mathcal {D} _ {s} |} \sum_ {\left(\boldsymbol {x} _ {i}, y _ {i}\right) \in \mathcal {D} _ {s}} \mathcal {L} \left(f \left(\boldsymbol {x} _ {i} + \boldsymbol {\delta} _ {i}; \boldsymbol {w}\right), y _ {i}\right). \tag {28}
$$
According to the gradient-matching technique [60, 61], we have the Upper-level Sub-problem as:
$$
\max  _ {\delta \subset \mathcal {B}} \frac {\nabla_ {\boldsymbol {w}} \mathcal {L} _ {t} \cdot \nabla_ {\boldsymbol {w}} \mathcal {L} _ {i}}{\| \nabla_ {\boldsymbol {w}} \mathcal {L} _ {t} \| \cdot \| \nabla_ {\boldsymbol {w}} \mathcal {L} _ {i} \|}, \tag {29}
$$
where we aim to maximize the gradient matching degree between  $\nabla_{\boldsymbol{w}}\mathcal{L}_t$  and  $\nabla_{\boldsymbol{w}}\mathcal{L}_i$  using cosine  $(\cdot)$  similarity as the metric through optimizing  $\delta$ . We solve the above upper-level subproblem via projected gradient ascend. We here use calculate  $\mathbb{E}_{(\hat{\boldsymbol{x}},y)\sim \mathcal{T}}[\mathcal{L}(f(\hat{\boldsymbol{x}};\boldsymbol{w}),y)]$  following:
$$
\mathbb {E} _ {(\hat {\boldsymbol {x}}, y) \sim \mathcal {T}} [ \mathcal {L} (f (\hat {\boldsymbol {x}}; \boldsymbol {w}), y) ] = \frac {1}{N} \sum_ {(\boldsymbol {x}, y) \in \mathcal {D}} \mathcal {L} (f (G _ {d} (\boldsymbol {x}); \boldsymbol {w}), y). \tag {30}
$$
Regarding the lower-level sub-problem, we have:
$$
\min  _ {\boldsymbol {w}} \left[ \frac {1}{| \mathcal {D} _ {s} |} \sum_ {\left(\boldsymbol {x} _ {i}, y _ {i}\right) \in \mathcal {D} _ {s}} \mathcal {L} \left(f \left(\boldsymbol {x} _ {i} + \boldsymbol {\delta} _ {i}; \boldsymbol {w}\right), y _ {i}\right) + \frac {1}{| \mathcal {D} _ {b} |} \sum_ {\left(\boldsymbol {x} _ {j}, y _ {j}\right) \in \mathcal {D} _ {b}} \mathcal {L} \left(f \left(\boldsymbol {x} _ {j}; \boldsymbol {w}\right), y _ {j}\right) \right]. \tag {31}
$$
After obtaining the poisoned dataset (i.e.,  $\mathcal{D}_s\cup \mathcal{D}_b$ ), we can optimize the model (i.e., ResNet-18) parameters  $\pmb{w}$  via solving the above Lower-level Upper-sub problem. The above Lower-level Upper-sub problem is solved via stochastic gradient descent.
We optimize the Upper-level and Lower-level Sub-problems alternatively for each optimization iteration. Specifically, we first train the model under benign dataset  $\mathcal{D}$ . Then for each iteration, we first optimize the Upper-level Sub-problem based on the trained model and obtain the perturbation  $\delta$ . After that, we optimize the Lower-level Sub-problem based on the obtained poisoned dataset. During each iteration for optimizing the above bi-level optimization problems, we optimize the Upper-level Sub-problem with 50 iterations, and optimize the Lower-level Sub-problem with 100 iterations. We optimize the entire bi-level optimization with five epochs. The other details for optimization hyperparameters as well as configuration are consistent with [61, 29].
In particular, to ensure the effectiveness of solving the aforementioned bi-level optimization problem, we have two additional strategies, as follows:
Domain I
Domain II
Domain III
Figure 10: The example of samples generated from various domain.
Domain IV
Domain V
- Strategy 1: Instead of randomly selecting samples from benign dataset  $\mathcal{D}$ , we here choose to select training samples with the largest gradient norms, following the previous work [60].
- Strategy 2: Instead of selecting samples from all classes, we follow the previous work [4] to select those from a specific class and the selected class is set as the target label. This strategy can enhance the effectiveness for solving the above bi-level optimization problem while preserving the verification performance for our approach.
C.2 The Process of Generating Samples from Other Domains
In this part, we describe how to generate samples from other domains (i.e.,  $(\overline{x},y)\sim \overline{T})$
After obtaining the transformation module  $G_{d}(\cdot)$ , we can generate hard-generalized domain samples from a specific domain. We here propose to generate samples from other domains by setting different configurations of  $\{w_i\}_1^4$ . For example, we can generate samples from the other domain by sampling  $\{w_i\}_1^4$  with another values following  $w_i \sim N(0,1)$ .
We here show some demonstration of samples from other domains in Fig. 10.
We here generated samples from other domains, and estimate  $\mathbb{E}_{(\overline{\boldsymbol{x}},y)\sim \overline{\mathcal{T}}}[\mathcal{L}(f(\overline{\boldsymbol{x}};\boldsymbol {w}),y)]$  following:
$$
\mathbb {E} _ {(\overline {{x}}, y) \sim \bar {\mathcal {T}}} [ \mathcal {L} (f (\overline {{x}}; \boldsymbol {w} (\delta)), y) ] = \frac {1}{N} \frac {1}{J} \sum_ {j} \sum_ {(\overline {{x}}, y) \in \bar {\mathcal {T}} _ {j}} \mathcal {L} (f (\overline {{x}}; \boldsymbol {w}), y), \tag {32}
$$
where  $\overline{T}_j$  represents the  $i$ -th unseen domain generated by the above approach.
C.3 The Selection of Hyper-parameters
After generating other unseen domains  $\mathcal{T}$ , we here describe the selection of hyper-parameters (i.e.,  $J$  and  $\lambda_3$ ) for generating protected dataset.
We here propose a heuristic approach for selecting  $J$  and  $\lambda_3$ . Specifically, we first keep  $\lambda_3$  fixed (i.e., 1) and adjust  $J$ . We conduct empirical study on CIFAR-10 tasks, the results are shown in Fig. 11.
We use ResNet-18 as the evaluated model. We generate several unseen domains using the above approach. We randomly select  $J$  of these domains for optimizing the Eq. (7), and select 3 unseen domains as the validation data. Notably, the validation domains are ensured visually different from the domains used for optimization.
From Fig. 11, we find that using  $\geq$  three unseen domains is sufficient to constrain the generalization performance for validation unseen domains. Therefore, we set  $J$  as 3 for our approach.
After that, we keep  $J$  fixed, and adjust  $\lambda_3$  gradually, the results are shown in Fig. 12. We find that when  $\lambda_3$  becomes smaller, the constraint for performance on other unseen domains reduces. Accordingly, we set  $\lambda_3$  as 0.3 for our approach since it can achieve a close generalization capacity compared to the benign DNN model (i.e., 24.3%).
Figure 11: Effects of the number of unseen Domains  $J$ .
Figure 12: Effects of the  $\lambda_{3}$
D The Proof for Theorem 2
$$
\sqrt {m - 1} \cdot (V - \eta + \tau) - t _ {\alpha} \cdot \sqrt {V - V ^ {2}} > 0, \tag {33}
$$
Proof. Since  $P_{b} > \eta$ , the original hypothesis  $H_{1}$  can be converted to
$$
H _ {1} ^ {\prime}: P _ {d} > \eta - \tau . \tag {34}
$$
Let  $E$  indicates the event of whether the suspect model  $f$  predicts a watermark sample as its ground-truth label  $y$ . As such,  $E \sim B(1,p)$ , where  $p = \operatorname{Pr}(C(\mathbf{X}') = Y)$  indicates the verification success probability and  $B$  is the Binomial distribution [41].
Let  $\hat{\pmb{x}}_1,\dots ,\hat{\pmb{x}}_m$  denotes  $m$  domain-watermarked samples used for dataset verification and  $E_{1},\dots ,E_{m}$  denote their prediction events, we know that the verification success rate  $V$  satisfies
$$
V = \frac {1}{m} \sum_ {i = 1} ^ {m} E _ {i}, \tag {35}
$$
$$
V \sim \frac {1}{m} B (m, p). \tag {36}
$$
According to the central limit theorem [41], the verification success rate  $V$  follows Gaussian distribution  $\mathcal{N}(p, \frac{p(1 - p)}{m})$  when  $m$  is sufficiently large. Similarly,  $(P_d - \eta + \tau)$  also satisfies Gaussian distribution. Accordingly, we can construct the t-statistic as follows:
$$
T \triangleq \frac {\sqrt {m} (W - \eta + \tau)}{s} \sim t (m - 1), \tag {37}
$$
where  $s$  is the standard deviation of  $(V - \eta + \tau)$  and  $V$ , i.e.,
$$
s ^ {2} = \frac {1}{m - 1} \sum_ {i = 1} ^ {m} (E _ {i} - V) ^ {2} = \frac {1}{m - 1} (m \cdot V - m \cdot V ^ {2}). \tag {38}
$$
To reject the hypothesis  $H_0$  at the significance level  $\alpha$ , we need to ensure that
$$
\frac {\sqrt {m} (V - \eta + \tau)}{s} > t _ {\alpha}, \tag {39}
$$
Figure 13: The example of domain watermark for CIFAR-10.
where  $t_{\alpha}$  is the  $\alpha$ -quantile of t-distribution with  $(m - 1)$  degrees of freedom.
According to equation (38)-(39), we have
$$
\sqrt {m - 1} \cdot (V - \eta + \tau) - t _ {\alpha} \cdot \sqrt {V - V ^ {2}} > 0. \tag {40}
$$
E The Detailed Settings for Experimental Datasets and Configurations
We evaluate our approach on three benchmark datasets (i.e., CIFAR-10 [1], Tiny-ImgaeNet [42], and STL-10 [45]). We here describe each benchmark dataset in detail.
CIFAR-10. CIFAR-10 dataset contains 10 labels, 50,000 training samples, and 10,000 validation samples. The training and validation samples are distributed evenly across each label. Each sample is resized as  $32 \times 32$  by default.
Tiny-ImageNet. Tiny-ImageNet dataset contains 200 labels, 100,000 training samples, and 10,000 validation samples. The training and validation samples are distributed evenly across each label. Each sample is resized as  $64 \times 64$  by default.
STL-10. STL-10 dataset contains 10 labels and 13,000 labeled samples and 100,000 unlabeled samples. We divide the labeled samples into the training and validation dataset with a ratio of  $8:2$ . The training and validation samples are distributed evenly across each label. Each sample is resized as  $96 \times 96$  by default.
Figure 14: The example of domain watermark for Tiny-ImageNet.
Table 4: Summary of accuracy (%) on samples from different domains for normal models and ours.
Task	Source domain	Target domain	Other domain
Normal	Ours	Normal	Ours	Normal	Ours
CIFAR-10	91.89	90.86	13.10	90.45	15.10	10.30
STL-10	85.61	84.58	9.50	82.00	16.00	11.60
Tiny-/ImageNet	60.13	59.10	6.00	58.08	12.60	15.40
E.2 The Demonstration of Domain Watermark for Each Dataset
We here show the domain watermark used for evaluating the effectiveness of our approach in the experiments. The demonstrations are shown in Figure 13, Figure 14, and Figure 15 for CIFAR-10, Tiny-ImageNet, and STL-10 datasets, respectively.
E.3 Training Configurations
In the experiments, we train each model with 150 epochs with an initialized learning rate of 0.1. Following previous work [29, 62], we schedule learning rate drops at epochs 14, 24, and 35 by a factor of 0.1. For all models, we employ SGD with Nesterov momentum, and we set the momentum coefficient to 0.9. We use batches of 128 images and weight decay with a coefficient of  $4 \times 10^{-4}$ . For each run, we report the verification success rate (VSR) averaged over the last 10 epochs when the models' accuracy converges. We report the results for each approach averaged over 5 runs.
Figure 15: The example of domain watermark for STL-10.