SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our: I like that it uses natural ingredients, and it was also good.
Src: This case is cute however this is the only good thing about it.
ChatGPT-p1: This case is not cute; however, it is the only good thing about it.
ChatGPT-p2: This case is not cute at all; however, it is the only bad thing about it.
Ours: This case is cute and overall a valuable product.
Table 6: A Sentiment transfer example, on which ChatGPT fails to completely reverse the overall sentiment of the sentence, although it successfully negates individual words within text. In contrast, our method achieves the sentiment reversal with minimal changes. ChatGPT-p1 and ChatGPT-p2 represent results obtained from two different prompts, i.e., p1: “Flip the sentiment of the following sentences, but keep the content unchanged as much as possible.”; p2: “Please invert the sentiment while preserving content as much as possible in the following sentence that originates from the original domain.”
6.4 Visualization of style variable
We further validate our theoretical insights within additional content-style disentanglement scenarios. As tense has a relatively sparse influence on sentences compared to their content, we choose tense (past and present) as another style for illustration. Specifically, we collect 1000 sentences in either past or present tense from the Yelp Dev set and derive their style representations, denoted as s, by feeding these sentences into our well-trained model. The projection results of CPVAE and MATTE are shown in 5. The distinct separation between the red and blue data points indicates a more discriminative and better
Figure 5: The style variables of sentences in past-tense (blue) and present-tense (red) following a UMAP projection. Left: CPVAE; Right: MATTE.
disentangled style variabl. However, in the case of CPVAE, some blue data points are mixed within the lower portion of the red region.
7 Conclusion and limitations
Prior work [Kong et al., 2022, Xie et al., 2023] have employed multiple domains to achieve unsupervised representation disentanglement. However, the assumed independence between the content and style variables often does not hold in real-world data distributions, particularly in natural languages. To tackle this challenge, we address the identification problem in latent-variable models by leveraging the sparsity structure in the data-generating process. This approach provides identifiability guarantees for both the content and the style variables. We have implemented a controllable text generation method based on these theoretical guarantees. Our method outperforms existing methods on various large-scale style transfer benchmark datasets, thus validating our theory. It is important to note that while our method shows promising empirical results for natural languages, the sparsity assumption (Assumption 1-iii.) may not hold for certain data distributions like images, where the style component could exert dense influences on pixel values. In such cases, we may explore other forms of inherent sparsity in the given distribution, e.g., sparse dependencies between content and style or sparse changes over multiple domains, to achieve identifiability guarantees and develop empirical approaches accordingly.
We thank anonymous reviewers for their constructive feedback. This work was funded by the UK Engineering and Physical Sciences Research Council (grant no. EP/T017112/1, EP/T017112/2, EP/V048597/1, EP/X019063/1). YH is supported by a Turing AI Fellowship funded by the UK Research and Innovation (grant no. EP/V020579/1, EP/V020579/2). The work of LK and YC is supported in part by NSF under the grants CCF-1901199 and DMS-2134080. This project is also partially supported by NSF Grant 2229881, the National Institutes of Health (NIH) under Contract R01HL159805, a grant from Apple Inc., a grant from KDDI Research Inc., and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
A. J. Bell and T. J. Sejnowski. An information-maximization approach to blind separation and blind deconvolution. Neural computation, 7(6):1129-1159, 1995.
N. Calderon, E. Ben-David, A. Feder, and R. Reichart. DoCoGen: Domain counterfactual generation for low resource domain adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7727-7746, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.533. URL https://aclanthology.org/2022.acl-long.533.
Y.-L. Chou, C. Moreira, P. Bruza, C. Ouyang, and J. Jorge. Counterfactuals and causability in explainable artificial intelligence: Theory, algorithms, and applications. Information Fusion, 81: 59-83, 2022.
P. Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.
N. Dai, J. Liang, X. Qiu, and X. Huang. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5997-6007, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1601. URL https://aclanthology.org/P19-1601.
S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, and R. Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=H1edEyBKDS.
M. Deng, B. Tan, Z. Liu, E. Xing, and Z. Hu. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580-7605, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.599. URL https://aclanthology.org/2021.emnlp-main.599.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
Q. Diao, M. Qiu, C.-Y. Wu, A. J. Smola, J. Jiang, and C. Wang. Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars). In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 193-202, 2014.
H. M. Dolatabadi, S. M. Erfani, and C. Leckie. Invertible generative modeling using linear rational splines. In S. Chiappa and R. Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 4236-4246. PMLR, 2020. URL http://proceedings.mlr.press/v108/dolatabadi20a.html.
P. Xu, J. C. K. Cheung, and Y. Cao. On variational learning of controllable representations for text without supervision. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10534-10543. PMLR, 2020. URL http://proceedings.mlr.press/v119/xu20a.html.
R. Xu, T. Ge, and F. Wei. Formality style transfer with hybrid textual annotations. ArXiv, abs/1903.06353, 2019b.
K. Yang and D. Klein. FUDGE: controlled text generation with future discriminators. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tür, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 3511-3535. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.naacl-main.276. URL https://doi.org/10.18653/v1/2021.naacl-main.276.
X. Yang, Y. Wang, J. Sun, X. Zhang, S. Zhang, Z. Li, and J. Yan. Nonlinear ICA using volume-preserving transformations. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=AMpki9kp8Cn.
Z. Yang, Z. Hu, C. Dyer, E. P. Xing, and T. Berg-Kirkpatrick. Unsupervised text style transfer using language models as discriminators. Advances in Neural Information Processing Systems, 31, 2018.
X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=Wo1HF2wWNZb.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process, 2022.
In this section, we first introduce the dataset. We then provide the network architecture details of MATTE. The hyperparameter selection criteria and the training details are summarized.
A.1.1 Dataset
We use four domain datasets to train our unsupervised model, i.e., Imdb, Yelp, Amazon and Yahoo, and follow the data split provided by Li et al. [2019]. The datasets can be downloaded via https://github.com/cookielee77/DAST. The dataset details can be found in Table 1. We set the sequence length  $L$  as 25, which is the 90 percentile of the sentence length of the training dataset. Therefore, shorter sentences are padded and longer sentences are clipped. The vocabulary size is set to 10000. For the sentiment transfer, we collect 100 positive sentences in dev set based on their sentiment labels to derive  $s_{\text{transfer}}$  to flip the sentiment of the negative sentences in the test set, and vice versa. For the tense transfer, we use stanfordnlp tool to identify the tense for the main verb, then collect 100 sentences of present tense in dev set to derive  $s_{\text{transfer}}$  to flip the past-tense sentences.
A.1.2 Model Architecture
We summarize our network architecture below and describe it in detail in Table 7.
Encoder: According to Xu et al. [2020], the encoder is fed with a text span  $\mathbf{x}[t_1:t_1 + m]$  extracted from the original sentence  $\mathbf{x}$ , where  $t_1$  is a random word position index,  $m$  is set to 10 if  $t_1 + m$  is smaller than  $L$ .  $\mathrm{H}_{\mathrm{word}}$  is the word embedding dimension, set to 256.  $\mathrm{H}_{\mathrm{lstm}}$  is the hidden states of LSTM, set to 1024.  $\mathrm{H}_{\mathbf{z}}$  is the dimension of the latent variable, set to 80. The output of the encoder is the  $\mu, \sigma$  and  $\mathbf{z}$ . All of them are in shape  $[\mathrm{BS},\mathrm{H}_{\mathbf{z}}]$ .
Decoder: Decoder is fed with the input sentence span and the generated latent variable. The final reconstructed sentence span is one timestamp delay compared to the input span, i.e.,  $\mathbf{x}_{t_1 + 1:(t_1 + 1) + m}$ . This is generated by applying beam search to the sequence of output probability over the vocabulary  $V$ . The  $\mathcal{L}_{\mathrm{recon}}$  is to calculate the cross-entropy loss between the output probability and target sequence span.
Content Flow  $r_c$ : We apply Deep Dense Sigmoid Flow (DDSF) [Huang et al., 2018] to derive the content noise term. To incorporate the domain information, we leverage the domain embedding (after MLP) to parameterize the flow model.
Style Flow  $r_s$ : We apply spline flow [Durkan et al., 2019] to derive the noise term. Similarly, we use the conditional flow with extra input. The conditional input is the combination of content variable and domain embedding. Specially, they are concatenated firstly and the result are fed into a MLP with Tanh activation to derive a attention score  $\alpha$ . The conditional input is actually the docPtrdcut.
A.1.3 Training
Training details. The models were implemented in PyTorch 2.0. and Python 3.9. The VAE network is trained for a maximum of 25 epochs and a mini-batch size of 64 is used. We use early stops if the validation reconstruction loss does not decrease for three epochs. For the encoder, we use the Adam optimizer and the learning rate of 0.001. For the decoder, we use SGD with a learning rate of 0.1. For the content and style flow, we use Adam optimizer and the learning rate is 0.001. We set three different random seeds and report the average results.
Training objective. The VAE-based model is mainly trained with  $\mathcal{L}_{\mathrm{recon}}$  and  $\mathcal{L}_{\mathrm{VAE}}$ . We use a training trick to better jointly train the other three objectives. The  $\mathcal{L}_{\mathrm{sparsity}}$  could cram the information of s to c, while the  $\mathcal{L}_{\mathrm{c - mask}}$  is used to prevent the ill-posed situation where s have zero influence. Therefore,
<sup>8</sup>https://stanfordnlp.github.io/stanfordnlp/
The implementation refers to ConditionedSpline in https://docs.pyro.ai/en/stable/_modules/pyro/distributions/transforms/spline.html
Module	Description	Output
1. Encoder	Encoder for Input sentence
Input xt1:t1+m	random span of sentence
WordEmb	get word embedding	BS × m × Hword
Bi-LSTM	Bi-direction, 2layers	BS × m × Hlstm
Average Pooling	sentence-level Rep.	BS × Hlstm
MLP	μ and σ	BS × (2 · Hz)
reparameterization	Sampling	BS × Hz
2. Domain Embedding	Embedding Layer
Input u	number of domain → udim	BS × udim
3. Content Flow rc
Input: c, u	domain as flow conditional input
MLP	u → conditional context	BS × |Hrc|
DDSF	get content noise term ā	BS × cdim
4. Style Flow rs
Input: c, u, s	content and domain as flow context
Concatenate	combine c and u	BS × (cdim + udim)
MLP	Tanh activation, get attention score α	BS × cdim
Element-wise Multiplication	α ⊙ c	BS × cdim
SplineFlow	get style noise term ās	BS × sdim
5. Decoder rs
Input: z, xt1:t1+m	generate the next token
Bi-LSTM	Bi-direction, 2layers	BS × m × Hlstm
MLP	output word probability	BS × m × V
Table 7: MATTE overall architecture. DDSF is deep dense sigmoid flow, and SplineFlow is neural spline flow.  $m$  is the length of randomly extracted text span from input sentence  $\mathbf{x}$  .
we involve both  $\mathcal{L}_{\mathrm{sparsity}}$  and  $\mathcal{L}_{\mathrm{c - mask}}$  at the beginning of the training phrase. For  $\mathcal{L}_{\mathrm{partial}}$ , it is used to sparsify the influence intersection but their separate influences change very frequently in the initial training stages. So we involve it after 3 epochs.
Computing hardware and running time. We used a machine with the following CPU specifications: AMD EPYC 7282 CPU. We use NVIDIA GeForce RTX 3090 with 24GB GPU memory. It costs approximately 190ms to run our model on this machine per epoch.
A.2 Additional Results
This section presents additional results on the hyperparameter sensitivity and the ablation studies on more datasets.
A.2.1 Hyperparameter Sensitivity
We discuss the effect of the three loss weights  $\lambda_{\mathrm{sparsity}}, \lambda_{\mathrm{partial}}$  and  $\lambda_{\mathrm{c-mask}}$  in the training objective. We have performed a grid search of  $\lambda_{\mathrm{sparsity}} \in [1\mathrm{E}-4,1\mathrm{E}-3,1\mathrm{E}-2]$ ,  $\lambda_{\mathrm{partial}} \in [3\mathrm{E}-5,3\mathrm{E}-3,3\mathrm{E}-1]$  and  $\lambda_{\mathrm{c-mask}} \in [1\mathrm{E}-4,1\mathrm{E}-3,1\mathrm{E}-2]$ . The best configuration is  $[\lambda_{\mathrm{sparsity}}, \lambda_{\mathrm{partial}}, \lambda_{\mathrm{c-mask}}] = [1\mathrm{E}-4,3\mathrm{E}-3,1\mathrm{E}-4]$ . The model performance is relatively sensitive to  $\lambda_{\mathrm{sparsity}}$ , so we plot the sentiment accuracy and BLEU as a function of  $\lambda_{\mathrm{sparsity}}$  in Figure 6.
Figure 6: The Sentiment Acc (left) and BLEU (right) with different  $\lambda_{\mathrm{sparsity}}$ . Among the four datasets, sentiment acc generally decreases as the  $\lambda_{\mathrm{sparsity}}$  becomes larger; BLEU increases instead. This observation aligns with our content identifiability theory. We determine the  $\lambda_{\mathrm{sparsity}}$  with the best G-score, i.e., 1E-4.
A.2.2 Ablation Results on Amazon and Yahoo Datasets
We show the ablation study of the Amazon and Yahoo datasets in Table 8. The full model achieves the best G-score and PPL on the two datasets. CausalDep improves the BLEU and PPL.  $\lambda_{\mathrm{sparsity}}$  greatly improves the content preservation at the cost of sentiment acc. After incorporating the  $\mathcal{L}_{\mathrm{partial}}$  and  $\mathcal{L}_{\mathrm{c - mask}}$ , the sentiment acc is recovered.
Amazon	Yahoo
Model	Acc(↑)	BLEU(↑)	G-score(↑)	PPL(↓)	Acc(↑)	BLEU(↑)	G-score(↑)	PPL(↓)
Backbone	32.60	41.08	30.08	77.61	43.92	25.44	20.28	76.28
Indep [Kong et al., 2022]	48.80▲	39.50	31.76▲	77.95	51.70	23.44	21.12▲	56.95▲
CausalDep	33.50▲	45.25▲	32.54▲	66.98▲	41.50	31.55▲	21.39▲	64.29▲
:w./Lsparsity	27.10	62.73△	34.73△	63.37△	27.32	48.21△	25.16△	60.03△
:w./Lpartial	33.10	58.54△	34.04	64.42	38.12△	41.74	28.03△	58.04△
:w./Lc-mask(Full)	34.50△	52.25	35.73△	63.37△	38.45△	42.40△	29.01△	56.12△
Table 8: Ablation results on sentiment transfer on two domains. CausalDep incorporates style flow  $r_s$  to model dependency of c on s, while Indep assumes the independence between the two variables. ▲ marks the improvements overBackbone, while △ over the CausalDep.
A.3 Semantic Preservation Measurement by CTC score
As BLEU has limitations in capturing semantic relatedness beyond literal word-level overlap, we adopt CTC score [Deng et al., 2021] as a complementary evaluation for semantics preservation measurement. For the semantics alignment from  $a$  to  $b$ , CTC considers the matching embeddings, i.e., maximum cosine similarity of all the tokens in  $a$  with the tokens in  $b$ , and vice versa. Then, the final semantic preservation is in  $F1$ -style definition with one direction result as precision, and the other one as recall. The evaluation results of all the baselines and MATTE are shown in Table 9. The CTC score still favours Optimus and MATTE, with most inferior results on  $\beta$ -VAE, which are similar trends under the BLEU evaluation schema. Admittedly, the CTC score differences are less discriminative than BLEU – this phoneme is also observed in Liu et al. [2022].
A.4 Diversity measurements for generated sentences
To further demonstrate the generation degradation issue—generate oversimplified and repetitious sentences, we use diversity-2 [Li et al., 2016], the ratio of distinct two-grams in all the two-grams in the generated sentences to evaluate the transferred sentences. The diversity-2 for original sentences is also included for better comparison. The results in Table 10 show that all the other methods except for  $\beta$ -VAE generated sentences with similar diversity-2 as the original sentences, but the sentences generated by  $\beta$ -VAE have much lower diversity than the original ones.
IMDB	Yelp	Amazon	Yahoo
BGST	0.468	0.458	0.472	0.458
β-VAE	0.436	0.433	0.433	0.413
JointTrain	0.456	0.462	0.455	0.437
CPVAE	0.462	0.463	0.461	0.443
GPT2-FT	0.459	0.459	0.458	0.448
Optimus	0.465	0.468	0.465	0.446
Matte	0.465	0.464	0.466	0.452
Table 9: CTC score, a complementary evaluation for semantics preservation.  $\beta$ -VAE displays the least impressive performance, and Optimus and Matte exhibit the overall best results.
Dataset	IMDB (0.34)	Yelp (0.63)	Amazon (0.64)	Yahoo(0.44)
β-VAE	0.11	0.46	0.37	0.22
JointTrain	0.21	0.59	0.56	0.37
CPVAE	0.32	0.59	0.57	0.45
MATTE	0.32	0.62	0.61	0.45
Table 10: Diversity-2 for the transferred sentences. Diversity for the original sentences is included in the bracket for comparison.  $\beta$ -VAE has significantly fewer distinct 2-gram than original datasets. This results are consistent with evaluation results on BLEU.
A.5 Proof for Theorem 1
The original Assumption 1 and Theorem 1 are copied below for reference.
ii. For all  $i \in \{1, \ldots, d_x\}$ , there exist  $\{\mathbf{z}^{(\ell)}\}_{\ell=1}^{|G_{i,:}|}$  and  $\mathbf{T} \in \mathcal{T}$ , such that  $\text{span}(\{\mathbf{J}_g(\mathbf{z}^{(\ell)})_{i,:}\}_{\ell=1}^{|G_{i,:}|}) = \mathbb{R}_{\mathcal{G}_{i,:}}^{d_z}$  and  $[\mathbf{J}_g(\mathbf{z}^{(\ell)})\mathbf{T}]_{i,:} \in \mathbb{R}_{\hat{\mathcal{G}}_{i,:}}^{d_z}$ .
iii. For every pair  $(c_{j_c}, s_{j_s})$  with  $j_c \in [d_c]$  and  $j_s \in \{d_c + 1, \dots, d_z\}$ , the influence of  $s_{j_s}$  is sparser than that of  $c_{j_c}$ , i.e.,  $\| \mathcal{G}_{:, j_c} \|_0 > \| \mathcal{G}_{:, j_s} \|_0$ .
$$
\underset {p _ {\hat {c}}, p _ {\hat {s}}, \hat {g}} {\arg \min } \sum_ {j _ {\hat {s}} \in \left\{d _ {c} + 1, \dots , d _ {z} \right\}} \left\| \hat {\mathcal {G}} _ {:, j _ {\hat {s}}} \right\| _ {0} \quad \text {s u b j e c t t o :} p _ {\hat {\mathbf {x}}} (\mathbf {x}) = p _ {\mathbf {x}} (\mathbf {x}), \forall \mathbf {x} \in \mathcal {X}, \tag {3}
$$
Proof. We first define the notation  $\mathbf{z} = [\mathbf{c},\mathbf{s}]$  and the indeterminacy function:
$$
h := \hat {g} ^ {- 1} \circ g,
$$
which is an invertible function  $h: \mathcal{Z} \to \hat{\mathcal{Z}}$  as  $g$  is invertible by Assumption 1-i.. According to the chain rule, we have the following relation among the Jacobian matrices:
$$
\mathbf {J} _ {\hat {g}} (\hat {\mathbf {z}}) = \mathbf {J} _ {g} (\mathbf {z}) \mathbf {J} _ {h} ^ {- 1} (\mathbf {z}). \tag {8}
$$
We define the support notations as follows:
$$
\begin{array}{l} \mathcal {G} := \operatorname {s u p p} (\mathbf {J} _ {g} (\mathbf {z})) \\ \hat {\mathcal {G}} := \operatorname {s u p p} (\mathbf {J} _ {\hat {\mathcal {G}}} (\hat {\mathbf {z}})), \\ \mathcal {T} := \operatorname {s u p p} \left(\mathbf {J} _ {h} ^ {- 1} (\mathbf {z})\right). \\ \end{array}
$$
In the following, we will show that  $(j_c,j_{\hat{s}})\notin \mathcal{T}$  for any  $j_{c}\in \{1,\ldots ,d_{c}\}$  and  $j_{\hat{s}}\in \{d_c + 1,\dots ,d_c+$ $d_{s}\}$ . That is,  $[\mathbf{J}_h^{-1}(\mathbf{z})]_{j_c,j_{\hat{s}}} = 0$ , for any  $j_{c}\in \{1,\dots ,d_{c}\}$  and  $j_{\hat{s}}\in \{d_c + 1,\dots ,d_c + d_s\}$ , which implies that  $\mathbf{c}$  is not influenced by  $\hat{\mathbf{s}}$ .
Because of Assumption 1-ii., for any  $i \in \{1, \ldots, d_{v_1} + d_{v_2}\}$ , there exists  $\{\mathbf{z}^{(\ell)}\}_{\ell=1}^{|G_{i,:}|}$  such that  $\mathrm{span}(\{\mathbf{J}_g(\mathbf{z}^{(\ell)})_{i,:}\}_{\ell=1}^{|G_{i,:}|}) = \mathbb{R}_{G_{i,:}}^{d_z}$ .
Since  $\{\mathbf{J}_g(\mathbf{z}^{(\ell)})_{i,:}\}_{\ell = 1}^{|G_{i,:}|}$  forms a basis of  $\mathbb{R}_{\mathcal{G}_{i,:}}^{d_z}$ , for any  $j_0 \in \mathcal{G}_{i,:}$ , we can express canonical basis vector  $\mathbf{e}_{j_0} \in \mathbb{R}_{\mathcal{G}_{i,:}}^{d_z}$  as:
$$
\mathbf {e} _ {j _ {0}} = \sum_ {\ell \in \mathcal {G} _ {i, :}} \alpha_ {\ell} \cdot \mathbf {J} _ {g} \left(\mathbf {z} ^ {(\ell)}\right) _ {i,:}, \tag {9}
$$
where  $\alpha_{\ell}\in \mathbb{R}$  is a coefficient.
Also, following Assumption 1-ii., there exists a deterministic matrix  $\mathbf{T}$  where  $\mathbf{T}_{j_1,j_2} \neq 0$  iff  $(j_{1},j_{2}) \in \mathcal{T}$  and
$$
\mathbf {T} _ {j _ {0,:}} = \mathbf {e} _ {j _ {0}} ^ {\top} \mathbf {T} = \sum_ {\ell \in \mathcal {G} _ {i,:}} \alpha_ {\ell} \cdot \mathbf {J} _ {g} \left(\mathbf {z} ^ {(\ell)}\right) _ {i,:} \mathbf {T} \in \mathbb {R} _ {\hat {\mathcal {G}} _ {i,:}} ^ {d _ {z}}, \tag {10}
$$
where  $\in$  is because each element in the summation belongs to  $\mathbb{R}_{\hat{\mathcal{G}}_{i,:}}^{d_z}$ .
$$
\forall j \in \mathcal {G} _ {i,:}, \mathbf {T} _ {j,:} \in \mathbb {R} _ {\hat {\mathcal {G}} _ {i,:}} ^ {d _ {z}}
$$
Equivalently, we have:
$$
\forall (i, j) \in \mathcal {G}, \quad \{i \} \times \mathcal {T} _ {j;}; \subset \hat {\mathcal {G}}. \tag {11}
$$
As both  $\mathbf{J}_g$  and  $\mathbf{J}_{\hat{g}}$  are invertible,  $\mathbf{J}_h(\mathbf{z})$  is an invertible matrix and thus has a non-zero determinant. Expressing  $\mathbf{J}_h(\mathbf{z})$  with the Leibniz formulae gives: