SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {i} ^ {n} \log p _ {s _ {i} | \mathbf {u}} \left(s _ {i} | \mathbf {u}\right) + \log | \det  \left(\mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}})\right) | = \sum_ {i} ^ {n} \log p _ {\hat {s} _ {i} | \mathbf {u}} \left(\hat {s} _ {i} | \mathbf {u}\right). \tag {101}
$$
Then we take the derivatives of both sides of Eq. (101) w.r.t.  $\hat{\mathbf{s}}_k$  and  $\hat{\mathbf{s}}_v$  where  $k, v \in \{1, \dots, n\}$  and  $k \neq q$ . For brevity, we first define the following terms:
$$
h _ {i, (k)} ^ {\prime} := \frac {\partial s _ {i}}{\partial \hat {s} _ {k}}, \tag {102}
$$
$$
h _ {i, (k, v)} ^ {\prime \prime} := \frac {\partial^ {2} s _ {i}}{\partial \hat {s} _ {k} \partial \hat {s} _ {v}}, \tag {103}
$$
$$
\eta_ {i} ^ {\prime} \left(s _ {i}, \mathbf {u}\right) := \frac {\partial \log p _ {s _ {i} \mid \mathbf {u}} \left(s _ {i} \mid \mathbf {u}\right)}{\partial s _ {i}}, \tag {104}
$$
$$
\eta_ {i} ^ {\prime \prime} \left(s _ {i}, \mathbf {u}\right) := \frac {\partial^ {2} \log p _ {s _ {i} \mid \mathbf {u}} \left(s _ {i} \mid \mathbf {u}\right)}{\left(\partial s _ {i}\right) ^ {2}}. \tag {105}
$$
$$
\sum_ {i = 1} ^ {n} \left(\eta_ {i} ^ {\prime \prime} \left(s _ {i}, \mathbf {u}\right) \cdot h _ {i, (k)} ^ {\prime} h _ {i, (v)} ^ {\prime} + \eta_ {i} ^ {\prime} \left(s _ {i}, \mathbf {u}\right) \cdot h _ {i, (k, v)} ^ {\prime \prime}\right) + \frac {\partial^ {2} \log | \det  (\mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}})) |}{\partial \hat {s} _ {k} \partial \hat {s} _ {v}} = 0. \tag {106}
$$
$$
\begin{array}{l} \sum_ {i = n _ {c} + 1} ^ {n} \left(\left(\eta_ {i} ^ {\prime \prime} \left(z _ {i}, \mathbf {u} _ {j}\right) - \eta_ {i} ^ {\prime \prime} \left(z _ {i}, \mathbf {u} _ {0}\right)\right) \cdot h _ {i, (k)} ^ {\prime} h _ {i, (v)} ^ {\prime}\right) (107) \\ \left. + \left(\eta_ {i} ^ {\prime} \left(z _ {i}, \mathbf {u} _ {j}\right) - \eta_ {i} ^ {\prime} \left(z _ {i}, \mathbf {u} _ {0}\right)\right) \cdot h _ {i, (k, v)} ^ {\prime \prime}\right) = 0, (108) \\ \end{array}
$$
where  $j = 1,\ldots ,2n_{D}$ . Note that for  $i\in \{1,\dots ,n_I\}$ ,  $\mathbf{s}_i$  does not depend on  $\mathbf{u}$ . Thus, we have  $\eta_i^{\prime \prime}(z_i,\mathbf{u}_j) = \eta_i^{\prime \prime}(z_i,\mathbf{u}_{j'})$  and  $\eta_i'(z_i,\mathbf{u}_j) = \eta_i'(z_i,\mathbf{u}_{j'})$ ,  $\forall j,j'$ . Hence only sources dependent on  $\mathbf{u}$  (i.e.,  $\mathbf{s}_D$ ) remain in Eq. (107).
By considering each  $j \in \{1, \dots, 2n_D\}$  for  $\mathbf{u}_j$ , we have  $2n_D$  equations like Eq. (107), which constitute a linear system with a  $2n_D \times 2n_D$  coefficient matrix.
According to Assumption ii, the coefficient matrix of the linear system has full rank. Thus, the only solution of Eq. (107) is  $h_{i,(k)}' h_{i,(q)}' = 0$  and  $h_{i,(k,v)}'' = 0$  for  $i = n_c + 1, \dots, n$  and  $k, v \in \{1, \dots, n\}, k \neq v$ .
$$
\mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) = \left[ \begin{array}{c c} \mathbf {A} := \frac {\partial \mathbf {s} _ {I}}{\partial \hat {\mathbf {s}} _ {I}} & \mathbf {B} := \frac {\partial \mathbf {s} _ {I}}{\partial \hat {\mathbf {s}} _ {D}} \\ \hline \mathbf {C} := \frac {\partial \mathbf {s} _ {D}}{\partial \hat {\mathbf {s}} _ {I}} & \mathbf {D} := \frac {\partial \mathbf {s} _ {D}}{\partial \hat {\mathbf {s}} _ {D}} \end{array} \right]. \tag {109}
$$
Because  $h_{i,(k)}' h_{i,(v)}' = 0, k, v \in \{1, \dots, n\}, k \neq v$ , for each  $i = n_I + 1, \dots, n$ , there is at most one index  $r \in \{1, \dots, n\}$  s.t.  $h_{i,(r)}' \neq 0$ . Therefore, there is at most one non-zero entry in each row indexed by  $i = n_D + 1, \dots, n$  in the Jacobian matrix  $\mathbf{J}_{\mathbf{h}^{-1}}(\hat{\mathbf{s}})$ . Further, the invertibility of  $\mathbf{h}^{-1}(\cdot)$  necessitates  $\mathbf{J}_{\mathbf{h}^{-1}}$  to be full-rank which implies that there is exactly one non-zero component in each row of sub-matrices  $\mathbf{C}$  and  $\mathbf{D}$ .
Suppose that non-zero component lies in the sub-matrix  $\mathbf{C}$ , then  $\mathbf{s}_D$  is not dependent on  $\mathbf{u}$ . Thus, the only non-zero component must lie in  $\mathbf{D}$ . Because  $\mathbf{J_h}$  is of full-rank and  $\mathbf{C}$  is a zero sub-matrix,  $\mathbf{D}$  must have full rank. Hence,  $\hat{\mathbf{s}}_D$  must be a composition of a component-wise invertible transformation and a permutation of  $\mathbf{s}_D$ . Moreover, according to part of the proof of Theorem 4.2 in Kong et al. (2022) (Steps 1, 2, and 3), the submatrix  $B$  is zero if Assumption iii holds.
Now we have shown that, for the matrix  $\mathbf{J}_{\mathbf{h}^{-1}}$ , its submatrix  $D$  is a generalized permutation matrix and both submatrices  $B$  and  $C$  are zero-matrices). Because  $\mathbf{h}^{-1}$  is smooth and invertible, the sub-matrices of the corresponding positions of  $\mathbf{J}_{\mathbf{h}}$  have the same properties. Then, we need to show the identifiability for the remaining sources  $\mathbf{s}_I = (s_1,\dots ,s_{n_I})$ , i.e.,  $\hat{\mathbf{s}}_I$  is a permutation with component-wise invertible transformation of  $\mathbf{s}_I$ .
Let  $\mathbf{D}(\mathbf{s})$  represents a diagonal matrix and  $\mathbf{P}$  represent a permutation matrix. Thus, our goal is equivalent to show that  $\mathbf{J}_{\mathbf{h}}(\mathbf{s}):_{n_I,:n_I} = \mathbf{P}_I\mathbf{D}_I(\mathbf{s})$  or  $\mathbf{J}_{\mathbf{h}^{-1}}(\hat{\mathbf{s}}):_{n_I,:n_I} = \mathbf{J}_{\mathbf{h}^{-1}}(\hat{\mathbf{s}}):_{n_I,:n_I} = \mathbf{J}_{\mathbf{h}}(\mathbf{s})_{:n_I,:n_I}^{-1} = \mathbf{D}_I(\mathbf{s})^{-1}\mathbf{P}_I^{-1}$ . By using chain rule repeatedly, we have
$$
\begin{array}{l} \mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) = \mathbf {J} _ {\mathbf {f} \circ \mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) \\ = \mathbf {J} _ {\mathbf {f}} \left(\mathbf {h} ^ {- 1} (\hat {\mathbf {s}})\right) \mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) \tag {110} \\ = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) \mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}). \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) _ {:; n _ {I}} = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) \mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) _ {:; n _ {I}} \\ \stackrel {(\star)} {=} \mathbf {J} _ {\mathbf {f}} (\mathbf {s}): _ {:: n _ {I}} \mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}): _ {n _ {I}: n _ {I}}, \tag {111} \\ \end{array}
$$
where Eq.  $(\star)$  is directly from the result that all entries in  $\mathbf{C}$ , i.e., those in  $\mathbf{J}_{\mathbf{h}^{-1}}(\hat{s})_{n_I + 1:::n_I}$ , are zero. Therefore our goal is equivalent to show that
$$
\mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) _ {:; n _ {I}} = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) _ {:; n _ {I}} \mathbf {D} _ {I} (\mathbf {s}) ^ {- 1} \mathbf {P} _ {I} ^ {- 1}. \tag {112}
$$
$$
\mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) _ {:,: n _ {I}} = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) _ {:,: n _ {I}} \mathbf {T} (\mathbf {s}), \tag {113}
$$
$$
\operatorname {s p a n} \left\{\mathbf {J} _ {\mathbf {f}} \left(\mathbf {s} ^ {(\ell)}\right) _ {i,: n _ {I}} \right\} _ {\ell = 1} ^ {\left| \mathcal {F} _ {i,: n _ {I}} \right|} = \mathbb {R} _ {\mathcal {F} _ {i,: n _ {I}}} ^ {n}. \tag {114}
$$
Since  $\{\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})_{i,:n_I}\}_{\ell = 1}^{\left|\mathcal{F}_{i,:n_I}\right|}$  forms a basis of  $\mathbb{R}_{\mathcal{F}_{i,:n_I}}^{n_I}$ , for any  $j_0 \in \mathcal{F}_{i,:n_I}$ , we are able to rewrite the one-hot vector  $e_{j_0} \in \mathbb{R}_{\mathcal{F}_{i,:n_I}}^{n_I}$  as
$$
e _ {j _ {0}} = \sum_ {\ell \in \mathcal {F} _ {i,: n _ {I}}} \alpha_ {\ell} \mathbf {J} _ {\mathbf {f}} \left(\mathbf {s} ^ {(\ell)}\right) _ {i,: n _ {I}}, \tag {115}
$$
$$
\mathrm {T} _ {j _ {0,:}} = e _ {j _ {0}} \mathrm {T} = \sum_ {\ell \in \mathcal {F} _ {i,: n _ {I}}} \alpha_ {\ell} \mathbf {J} _ {\mathbf {f}} \left(\mathbf {s} ^ {(\ell)}\right) _ {i,: n _ {I}} \mathrm {T} \in \mathbb {R} _ {\hat {\mathcal {F}} _ {i,: n _ {I}}} ^ {n _ {I}}, \tag {116}
$$
where the final “ $\in$ ” follows from Assumption i that each element in the summation belongs to  $\mathbb{R}_{\hat{\mathcal{F}}_{i,:n_I}}^{n_I}$ . Thus
$$
\forall j \in \mathcal {F} _ {i,: n _ {I}}, \mathrm {T} _ {j,:} \in \mathbb {R} _ {\hat {\mathcal {F}} _ {i,: n _ {I}}} ^ {n _ {I}}. \tag {117}
$$
$$
\forall (i, j) \in \mathcal {F} _ {:,: n _ {I}}, \{i \} \times \mathcal {T} _ {j,:} \subset \hat {\mathcal {F}} _ {:,: n _ {I}}. \tag {118}
$$
$$
\det  (\mathbf {T} (\mathbf {s} ^ {(\ell)})) = \sum_ {\sigma \in \mathcal {S} _ {n I}} \left(\operatorname {s g n} (\sigma) \prod_ {i = 1} ^ {n _ {I}} \mathbf {T} (\mathbf {s} ^ {(\ell)}) _ {i, \sigma (i)}\right) \neq 0, \tag {119}
$$
where  $S_{n_I}$  is the set of  $n_I$ -permutations. Thus, there is at least one term in the sum that is non-zero, i.e.,
$$
\exists \sigma \in \mathcal {S} _ {n _ {I}}, \forall i \in \{1, \dots , n _ {I} \}, \operatorname {s g n} (\sigma) \prod_ {i = 1} ^ {n _ {I}} \mathbf {T} \left(\mathbf {s} ^ {(\ell)}\right) _ {i, \sigma (i)} \neq 0, \tag {120}
$$
$$
\exists \sigma \in \mathcal {S} _ {n _ {I}}, \forall i \in \{1, \dots , n _ {I} \}, \mathbf {T} (\mathbf {s} ^ {(\ell)}) _ {i, \sigma (i)} \neq 0. \tag {121}
$$
$$
\forall j \in \{1, \dots , n _ {I} \}, \sigma (j) \in \mathcal {T} _ {j,:}. \tag {122}
$$
Combined with Eq. (118), we have
$$
\forall (i, j) \in \mathcal {F} _ {:; n _ {I}}, (i, \sigma (j)) \in \{i \} \times \mathcal {T} _ {j,:} \subset \hat {\mathcal {F}} _ {:; n _ {I}}. \tag {123}
$$
$$
\sigma \left(\mathcal {F} _ {:; n _ {I}}\right) = \left\{\left(i, \sigma (j)\right) \mid (i, j) \in \mathcal {F} _ {:; n _ {I}} \right\}. \tag {124}
$$
$$
\sigma \left(\mathcal {F} _ {:; n _ {I}}\right) \subset \hat {\mathcal {F}} _ {:; n _ {I}}. \tag {125}
$$
$$
| \hat {\mathcal {F}} _ {:,: n _ {I}} | \leq | \mathcal {F} _ {:,: n _ {I}} | = | \sigma \left(\mathcal {F} _ {:,: n _ {I}}\right) |. \tag {126}
$$
Together with Eq. (125), it follows that
$$
\sigma \left(\mathcal {F} _ {:,: n _ {I}}\right) = \hat {\mathcal {F}} _ {:,: n _ {I}}. \tag {127}
$$
Suppose  $\mathbf{T}(\mathbf{s})\neq \mathbf{D}_I(\mathbf{s})^{-1}\mathbf{P}_I^{-1}$  ,then
$$
\exists j _ {1} \neq j _ {2}, \mathcal {T} _ {j _ {1,:}} \cap \mathcal {T} _ {j _ {2,:}} \neq \emptyset . \tag {128}
$$
$$
\sigma \left(j _ {3}\right) \in \mathcal {T} _ {j _ {1},:} \cap \mathcal {T} _ {j _ {2},:}. \tag {129}
$$
Since  $j_1 \neq j_2$ , we can assume  $j_3 \neq j_1$  without loss of generality. Based on Assumption iv, there exists  $\mathcal{C}_{j_1} \ni j_1$  such that  $\bigcap_{i \in \mathcal{C}_{j_1}} \mathcal{F}_{i,:n_I} = \{j_1\}$ . Because
$$
j _ {3} \notin \{j _ {1} \} = \bigcap_ {i \in \mathcal {C} _ {j _ {1}}} \mathcal {F} _ {i,: n _ {I}}, \tag {130}
$$
there must exist  $i_3 \in \mathcal{C}_{j1}$  such that
$$
j _ {3} \notin \mathcal {F} _ {i _ {3},: n _ {I}}. \tag {131}
$$
Since  $j_1 \in \mathcal{F}_{i_3:n_I}$ , we have  $(i_3,j_1) \in \mathcal{F}_{\cdot ;n_I}$ . Therefore, according to Eq. (118), we have
$$
\left\{i _ {3} \right\} \times \mathcal {T} _ {j _ {1},:} \subset \hat {\mathcal {F}} _ {:,: n _ {I}}. \tag {132}
$$
Note that  $\sigma(j_3) \in \mathcal{T}_{j_1,:} \cap \mathcal{T}_{j_2,:}$  implies
$$
\left(i _ {3}, \sigma \left(j _ {3}\right)\right) \in \left\{i _ {3} \right\} \times \mathcal {T} _ {j _ {1},..} \tag {133}
$$
Then by Eqs. (132) and (133), we have
$$
\left(i _ {3}, \sigma \left(j _ {3}\right)\right) \in \hat {\mathcal {F}} _ {:,: n _ {I}}. \tag {134}
$$
This implies  $(i_3,j_3)\in \mathcal{F}_{\therefore n_I}$  by Eqs. (124) and (127), which contradicts Eq. (131). Therefore, we have proven by contradiction that  $\mathbf{T}(\mathbf{s}) = \mathbf{D}_I(\mathbf{s})^{-1}\mathbf{P}_I^{-1}$ . By replacing  $\mathbf{T}(\mathbf{s})$  with  $\mathbf{D}_I(\mathbf{s})^{-1}\mathbf{P}_I^{-1}$  in Eq. (113), we obtain Eq. (112), which is the goal.
In this section, we describe the experimental settings as well as some additional results.
B.1 Supplementary experimental settings
To produce observational data that meets the required assumptions for different models, we simulate the sources and mixing process as follows:
UCSS. To ensure that the true nonlinear mixing process adheres to the Structural Sparsity condition (Assumption ii in Thm. 3.1), as per previous work (Zheng et al., 2022), we generate observed variables in a structured way: Each observed variable is only a nonlinear mixture of its direct ancestors. For instance, if the observed variable  $\mathbf{x}_1$  has parents  $\mathbf{s}_1$  and  $\mathbf{s}_2$ , then  $\mathbf{x}_1 = \mathbf{f}_1(\mathbf{s}_1, \mathbf{s}_2)$ . We use Generative Flow (GLOW) (Kingma and Dhariwal, 2018) with a projection layer as the nonlinear function  $\mathbf{f}_i$ . The difference between GLOW and GIN is that GLOW does not impose a constraint on the determinant of the Jacobian, thus being more suitable for the general nonlinear function since it has less inductive bias. The implementation of GLOW is a part of FrEIA<sup>1</sup> (Ardizzone et al., 2018-2022).
The ground-truth sources are sampled from a multivariate Gaussian, with zero means and variances sampled from a uniform distribution on  $[0.5, 3]$ , which are of the same values as in previous works (Khemakhem et al., 2020a; Sorrenson et al., 2020; Zheng et al., 2022). It is worth noting that we sample sources from a single multivariate Gaussian so that all sources are marginally independent, unlike from most previous works assuming conditional independence given auxiliary variables.
Mixed. For the Mixed model, we partition sources into  $\mathbf{s}_I$  and  $\mathbf{s}_D$ . For sources in  $\mathbf{s}_I$ , we sample them in the same way as that for UCSS. For sources in  $\mathbf{s}_D$ , we sample them from  $2n_D + 1$  multivariate Gaussian distributions as required by Assumption ii in Thm. 4.4. Similarly, these multivariate Gaussian distributions are of zero means and variances sampled from a uniform distribution on [0.5, 3]. For sources in  $\mathbf{s}_I$ , we generate their influences on the observed variables in a structured way described above to satisfy the partial sparsity assumption (Assumption iv in Thm. 4.4); for sources in  $\mathbf{s}_D$ , we remove the constraint on the structure and permit each source to affect all observed variables.
Base. For the Base model, following (Sorrenson et al., 2020), we use GLOW (Kingma and Dhariwal, 2018) as the mixing function to generate the data. The sources are from a single multivariate Gaussian distribution with zero means and variances uniformly sampled from the interval [0.5, 3]. No constraints on the structure have been imposed for the Base model.
In the evaluation of our model, we utilize the Mean Correlation Coefficient (MCC) as a metric for assessing the correspondence between the ground-truth and recovered latent sources. The MCC is calculated by first determining the pair-wise correlation coefficients between the true sources and the recovered sources after a nonlinear component-wise transformation learned by regression. Subsequently, an assignment problem is solved to match each recovered source with the corresponding ground-truth source that exhibits the highest correlation. MCC is a widely accepted metric in the literature for measuring the degree of identifiability, accounting for component-wise transformations (Hyvärinen and Morioka, 2016). Our results are all based on 20 trials, each with a different random seed.
For the synthetic datasets used in our experiments, the sample size is 2000. The parameters used for training include a learning rate of 0.01 and a batch size of 200. Additionally, the number of coupling layers for both GIN and GLOW is set as 10. In regard to the "Triangles" dataset, it comprises 60,000  $32 \times 32$  images of drawn triangles. The statistics of the dataset are described in (Yang et al., 2022). For the experiments conducted on this dataset, the learning rate is set at  $3 \times 10^{-4}$  and the batch size is 100. Concerning the EMNIST dataset, it includes 240,000  $28 \times 28$  images of real-world handwritten digits. The learning rate and batch size used for these experiments are  $3 \times 10^{-4}$  and 240, respectively. The experiments are conducted directly using the official implementation of  $\mathrm{GIN}^2$  (Sorrenson et al., 2020) with an additional sparsity regularization term on the Jacobian of the estimated mixing function.
$^{1}$ https://github.com/vislearn/FrEIA
$^{2}$ https://github.com/VLL-HD/GIN
B.2 Supplementary experimental results
In this section, we delve deeper into the applicability of our results by offering additional empirical studies that further illuminate the implications of the proposed theory. Specifically, we focus on the following aspects: (1) the effect of different regularization terms on the performance of identification; (2) the applicability of the theory as illustrated by additional real-world examples.
Regularization. For the regularization term, directly utilizing the  $\ell_0$  penalty may be computationally infeasible as it results in a discrete optimization problem. To overcome this issue, we adopt the  $\ell_1$  regularizer, which has been extensively studied in the literature for high-dimensional support recovery, particularly for variable selection (Wainwright, 2009) and Gaussian graphical model selection (Ravikumar et al., 2008). The usage of  $\ell_1$  regularizer induces sparsity in the solution; however, it may also introduce bias which can negatively affect the performance (Fan and Li, 2001; Breheny and Huang, 2011). This is because the  $\ell_1$  norm penalty also
Figure 8: MCC of UCSS w.r.t. different sparsity regularizations and numbers of sources.
penalizes both small and large entries, unlike the  $\ell_0$  norm which remains constant for nonzero entries. To remedy this bias issue, we explore alternative penalties, such as the smoothly clipped absolute deviation (SCAD) penalty (Fan and Li, 2001) and minimax concave penalty (MCP) (Zhang, 2010), which can be interpreted as hybrids of  $\ell_0$  and  $\ell_1$  penalties. Additionally, we note that the support recovery of the  $\ell_1$  penalty is based on the incoherence conditions in various cases (Wainwright, 2009; Ravikumar et al., 2008, 2011), which may be restrictive in practice, whereas the SCAD and MCP penalties do not rely on such conditions (Loh and Wainwright, 2017). Based on our experimental results (Fig. 8), we adopt the MCP penalty as the regularization term.
EMNIST. To further demonstrate the generalizability of the proposed identifiability results, we present identification results for all digits on the EMNIST dataset. As previously mentioned, we show the recovered attributes with the top-4 singular values. From Fig. 9, it is clear that these attributes are highly interpretable and appear to be the underlying concepts that influence the process of writing digits by hand. This indicates the potential applicability of our assumptions in real-world scenarios.
(a) Line thickness
(b) Angle
(c) Upper width
(d) Height
Figure 9: Results for all digit classes within the EMNIST dataset. We present the identified sources with the top-4 standard deviations SDs. Each sub-figure represents a source identified by our model, with its value varying from  $-4$  to  $+4$  SDs to illustrate its influence. The rightmost column presents a heat map given by the absolute pixel difference between the  $-1$  and  $+1$  SDs. The interpretation of these sources may correspond to line thickness, angle, upper width, and height, respectively.