SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As mentioned in the main body, we include a live demo for our TranSVAE framework. This demo can be accessed at: https://huggingface.co/spaces/1dkong/TranSVAE. Here we include the detailed instructions for playing with this demo.
Table A: Summary of notations used in this paper.
Notation	Description
D	Domain
S/T	Source domain / Target domain
V^D	A video sequence from domain D
V_i^D, y_i^D	The i-th video sequence and the corresponding action label of domain D
x_i^D, ..., x_T^D	T frames of images in the video sequence V^D
z_i^D, ..., z_T^D	The dynamic latent factors of the video sequence from domain D
z_d^D	The static latent factors of the video sequence from domain D
x<t^D	The input sequence before timestamp t
x1:T^D	The full input sequence from t = 1 to t = T.
Table B: Summary of the best-possible hyperparameter values of the TranSVAE framework for each UDA task in our experiments after an extensive grid search.
Task	Backbone	T	η	λ1	λ2	λ3	λ4
U → H	I3D	8	0.93	50	1	1	1
H → U	I3D	9	0.96	0.5	0.1	10	1
JS → JT	I3D	6	0.95	0.001	10	100	10
D1 → D2	I3D	9	0.96	50	10	5	100
D1 → D3	I3D	10	1	1	0.5	0.1	100
D2 → D1	I3D	8	0.93	100	10	5	100
D2 → D3	I3D	10	0.91	0.5	10	50	100
D3 → D1	I3D	8	0.91	100	10	50	100
D3 → D2	I3D	9	0.91	1000	1	0.1	100
This demo is built upon Hugging Face Spaces<sup>4</sup>, which provides concise and easy-to-use live demo interfaces. Our demo consists of one input interface and one output interface as shown in Fig. B. Specifically, the appearances of theSprites avatars are fully controlled by four attributes, i.e., body, hair color, top wear, and bottom wear. We construct two domains,  $\mathbf{P}_1$  and  $\mathbf{P}_2$ .  $\mathbf{P}_1$  uses the "Human" body while  $\mathbf{P}_1$  uses the "Alien" body. The attribute pools of "Human" and "Alien" are non-overlapping across domains, resulting in completely heterogeneous  $\mathbf{P}_1$  and  $\mathbf{P}_2$ . Each video sequence contains eight frames in total.
For conducting domain disentanglement and transfer with TranSVAE, users are free to choose the action and the appearance of the avatars on the left-hand side of the interface. Next, simply click the "Submit" button and the adaptation results will display on the right-hand side of the interface in a few seconds. The outputs include:
- The 1st column: The original input of the "Human" and "Alien" avatars, i.e.,  $\{\mathbf{x}_1^{\mathbf{P}_1},\dots,\mathbf{x}_8^{\mathbf{P}_1}\}$  and  $\{\mathbf{x}_1^{\mathbf{P}_2},\dots,\mathbf{x}_8^{\mathbf{P}_2}\}$ ;
- The 2nd column: The reconstructed "Human" and "Alien" avatars  $\{\widetilde{\mathbf{x}}_1^{\mathbf{P}_1},\dots,\widetilde{\mathbf{x}}_8^{\mathbf{P}_1}\}$  and  $\{\widetilde{\mathbf{x}}_1^{\mathbf{P}_2},\dots,\widetilde{\mathbf{x}}_8^{\mathbf{P}_2}\}$ ;
- The 3rd column: The reconstructed "Human" and "Alien" avatars using only  $\{\mathbf{z}_1^{\mathcal{D}},\dots,\mathbf{z}_8^{\mathcal{D}}\}$ ,  $\mathcal{D} \in \{\mathbf{P}_1, \mathbf{P}_2\}$ , which are domain-invariant;
<sup>4</sup>https://huggingface.co
TranSVAE0
encoder): encoder()
(c1): dcgan_conv
main): Sequential()
(0): Conv2d(3, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
(2):LeakyReLU(negative slope=0.2,inplace=True))
(c2): dcgan_cony( (main): Sequential)
(0): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
(2):LeakuRelU(negativeslope=0.2,inplace=True))
(c3): dcgan_cony( (main): Sequential)
(0): Cony2d(128, 256, kernel size=(4, 4), stride=(2, 2), padding=(1, 1))
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
(2):LeakuReLU(negative_slope=0.2,inplace=True))
(c4): dcgan_conv (main): Sequential
(0): Conv2d(256, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
(2):LeakyReLU(negative_slope=0.2,inplace=True))
(c5): Sequential
(0): Conv2d(512, 1024, kernel_size=(4, 4), stride=(1, 1))
(1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running stats=True)
(2):Tanh()
```python
(decoder): decoder.pySkip
(upc1): Sequential(
):ConvTranspose2d(1024,512,kernel_size=(4,4),stride=(1,1))
): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
:LeakyReLU(negative_slope=0.2, inplace=True))
(upc2): dcgan_upconv( (main): Sequential
10: ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
0: BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
LeakyReLU(negative_slope=0.2, inplace=True))
(upc3): dcgan_upconv( main): Sequential
10: ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
(upc4): dcqan_upconv( (main): Sequential
1): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
0: BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
(upc5): Sequential(
):ConvTranspose2d(64,3,kernel_size=(4,4),stride=(2,2)，padding=(1,1))
):Sigmoid()
(relu):LeakyReLU(negative_slope=0.1)
( dropout_f): Dropout(p=0.5, inplace=False)
( dropout_v): Dropout(p=0.5, inplace=False)
(z_bPHor_istm_ig1): LSTMCell(512, 512)
(zPrior_lstm_ly2): LSTMCell(512, 512)
(z_prior_mean): Linear(in_features=512, out_features=512, bias=True)
(z_prior_logvar): Linear(in_features=512, out_features=512, bias=True)
(z_lstm): LSTM(1024, 512, batch_first=True, bidirectional=True)
(f_mean): Linear(in_features=1024, out_features=512, bias=True)
(f_logvar): Linear(in_features=1024, out_features=512, bias=True)
(z_rnn): RNN(1024, 512, batch_first=True)
(z_mean): Linear(in_features=512, out_features=512, bias=True)
(z logvar): Linear(in features=512, out features=512, bias=True)
(fc_feature_domain_frame): Linear(in_features=512, out_features=512, bias=True)
(fc-classifier_domain_frame): Linear(in_features=512, out_features=2, bias=True)
(TRN)
bn_trn_S): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
bn_trn_T: BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running.stats=True)
(fc_feature_domain Video): Linear(in_features=256, out_features=256, bias=True)
(fc-classifier_domain Video): Linear(in_features=256, out_features=2, bias=True)
(relation domain classifier all):Module list(
(0): Sequential(
:Linear(in features=256,out features=256,bias=True)
D:ReLU0
Linear(in_features=256, out_features=2, bias=True))
(1): Sequential
D:RelU0
(2): Sequential
:Linear(in features=256 out features=256 bias=True)
:Linear(in features=256,out features=2,bias=True))
(3): Sequential
: Linear(in features=256, out features=256, bias=True)
:ReLU0
(4): Sequential
): Linear(in_features=256, out_features=2, bias=True))
(5): Sequential
): Linear(in_features=256, out_features=256, bias=True)
(6): Sequential
(pred_classification Video): Linear(in_features=256, out_features=15, bias=True)
(fc_feature_domain_long): Linear(in_features=512, out_features=512, bias=True)
(fc-classifier_doamin_forent): Linear(in_features=512, out_features=2, bias=True)
Figure A: The neural network structure (left) and a Netron graph (right) of the proposed TranSVAE framework. Zoom-ed in for the details.
Figure B: The input (left) and output (right) interfaces of our live demo. Users are free to customize the actions and appearances of the source and target inputs (i.e., the "Human" and "Alien" avatars) in the left-hand side and use them for disentanglement and transfer as shown in the right-hand side.
Table C: Ablation results without and with disentanglement term.
Lsvae	Lcls	Ladv	Lmi	Lctc	U → H	H → U
✓	✓	81.67	90.54
✓	✓	79.44	90.37
✓	✓	81.11	90.37
✓	✓	✓	✓	82.22	91.07
✓	✓	✓	84.44	93.52
✓	✓	✓	83.06	93.70
✓	✓	✓	83.61	91.42
✓	✓	✓	✓	✓	87.78	98.95
- The 4th column: The reconstructed "Human" and "Alien" avatars by exchanging  $\mathbf{z}_d^{\mathcal{D}}$ , which results in two sequences with the same actions but exchanged appearance, i.e., domain disentanglement and transfer.
In this section, we provide additional quantitative and qualitative results for our TranSVAE framework.
B.1 Additional Ablation Studies
Component Integration Study. We further analyze the effect of each loss term without disentanglement, i.e., not adding  $\mathcal{L}_{\mathrm{svae}}$ , on the UCF-HMDB dataset, as shown in Tab. C. Without  $\mathcal{L}_{\mathrm{svae}}$ , the framework is to learn representations with corresponding constraints as many existing video-based UDA methods do. Precisely, we have the following remarks for the top-half table:
- Only with  $\mathcal{L}_{\mathrm{adv}}$ , the framework is equivalent to  $\mathrm{TA}^3\mathrm{N}$  [7], which learns the domain-invariant temporal representation. We can see the adaptation results are similar to that of [7].
-  $\mathcal{L}_{\mathrm{mi}}$  is a term specifically designed for disentanglement purposes. Thus, the effect of using  $\mathcal{L}_{\mathrm{mi}}$  alone for adaptation is random as verified in the above table, achieving worse results
Table D: Ablation results for the frame number  $T$  .
Task	5	6	7	8	9	10	12	14	16	20
U → H	84.44	86.11	84.17	87.78	84.72	85.28	83.89	84.44	82.78	85.00
H → U	96.85	94.22	98.42	94.75	98.95	93.70	93.87	94.40	94.05	94.40
Table E: Ablation results for the pseudo-label threshold  $\eta$  .
Task	w/o	0.90	0.91	0.92	0.93	0.94	0.95	0.96	0.97	0.98	0.99
U → H	87.22 (−0.56)	86.94	87.22	87.50	87.78	86.11	86.67	86.94	86.39	86.11	86.39
H → U	94.40 (−4.55)	98.42	97.37	98.77	98.60	98.25	98.25	98.95	97.90	97.72	95.27
in the  $\mathbf{U} \to \mathbf{H}$  task and better results in the  $\mathbf{H} \to \mathbf{U}$  tasks compared with the source-only baseline.
-  $\mathcal{L}_{\mathrm{ctc}}$  enforces the learned static representation to be domain-specific and brings indirect benefit to adaptation, thus obtaining slightly better results than the source-only baseline.
In sum, without  $\mathcal{L}_{\mathrm{svae}}$ , the adaptation performance of each loss term alone is far from optimal. Moreover, combining all the terms can slightly improve the adaptation performance over the individual ones, however, is still obviously worse than TranSVAE results. The above ablation study shows the necessity of the disentanglement in the TranSVAE framework and proves that  $\mathcal{L}_{\mathrm{svae}}$  is an indispensable loss term.
Regarding the bottom half table showing the results with  $\mathcal{L}_{\mathrm{adv}}$ , we have the following remarks:
- Each individual term with  $\mathcal{L}_{\mathrm{adv}}$  achieves better adaptation results than the source-only baseline.
Each individual term with  $\mathcal{L}_{\mathrm{adv}}$  consistently outperforms the corresponding ones without  $\mathcal{L}_{\mathrm{adv}}$ .
- The final TranSVAE framework combined all the loss terms yields much better results than each individual case.
This set of ablation studies indicates that 1) each term in TranSVAE brings positive effects to adaptation, and 2) disentanglement and adaptation are interactive and complementary in our framework to obtain the best possible UDA results.
Number of Frames  $T$ . Tab. D shows the transfer performance with the variation of the number of frames  $T$  on the UCF-HMDB dataset. As can be seen, the two tasks achieve the optimal performance with different  $T$ , specifically  $T = 8$  for  $\mathbf{U} \rightarrow \mathbf{H}$  and  $T = 9$  for  $\mathbf{H} \rightarrow \mathbf{U}$ . Based on this observation, we apply a grid search on the validation set to obtain the optimal  $T$  for each task in our experiments.
Target Pseudo-Label Threshold  $\eta$ . We show the sensitivity analyses of the transfer performance with respect to the target pseudo label threshold  $\eta$  on the UCF-HMDB dataset in Tab. E. The results show that different tasks yield the best transfer result with different  $\eta$ , specifically  $\eta = 0.93$  for  $\mathbf{U} \to \mathbf{H}$  and  $\eta = 0.96$  for  $\mathbf{H} \to \mathbf{U}$ . Thus, we also apply the grid search on the validation set to obtain the optimal  $\eta$  for each task.
B.2 Additional Qualitative Results
For those who cannot access our live demo, we have included more qualitative examples for domain disentanglement and transfer in Fig. C. We also provide a link<sup>5</sup> for GIFs demonstrating various disentanglement and reconstruction results.
In the GIFs link, we show two cases, the first two columns for the first one and the last two columns for the second one. Each case contains a source and a target cartoon character performing an action. For each row, we have the following remark:
$^{5}$ https://github.com/ldkong1205/TranSVAE#ablation-study
Figure C: Additional qualitative results for illustrating the domain disentanglement and transfer properties in our TranSVAE framework.
- The second row shows the reconstructed results of TranSVAE, i.e.  $\mathbf{z}_d^{\mathcal{D}}$  and  $\mathbf{z}_t^{\mathcal{D}}$ . As can be seen, TranSVAE reconstructs the image with a high quality.
- The third row shows the reconstructed results only using the static latent factors, i.e.,  $\mathbf{z}_d^{\mathcal{D}}$  and  $\mathbf{0}_t^{\mathcal{D}}$ , where we replace  $\mathbf{z}_t^{\mathcal{D}}$  with zero vectors. As can be seen, the reconstructed results are basically static containing the appearance of the character which is the main domain difference in theSprites dataset [22]. Specifically, we find they generally lack arms. This is reasonable as the target action is slashing or spelling cards with arms moving, and such dynamic information on the arm is captured by  $\mathbf{z}_t^{\mathcal{D}}$ .
- The fourth row shows the reconstructed results only using the dynamic latent factors, i.e.,  $\mathbf{0}_d^{\mathcal{D}}$  and  $\mathbf{z}_t^{\mathcal{D}}$ , where we replace  $\mathbf{z}_d^{\mathcal{D}}$  with zero vectors. As can be seen, the reconstructed results are performing the right action but the appearance is mixed up. This shows that  $\mathbf{z}_t^{\mathcal{D}}$  are indeed domain-invariant and contain the semantic information.
- The last row shows the reconstructed results of exchanging the dynamic latent factors between domains, i.e.,  $(\mathbf{0}_d^S, \mathbf{z}_t^T)$  and  $(\mathbf{0}_d^T, \mathbf{z}_t^S)$ . As can be seen, the reconstructed results are with the original appearance but perform the transferred action. This indicates the potential of TranSVAE for some style-transfer tasks.
C Broader Impact
This paper provides a novel transfer method to use cross-domain video data, which effectively helps reduce the annotation efforts in related video applications. Although the main empirical evaluation is on the video action recognition task, the model structure proposed in this paper is also applicable to other video-related tasks, such as action segmentation, video semantic segmentation, etc. More generally, the idea of disentangling domain information sheds light on other data modality style transfer tasks, e.g., voice conversion. The negative impacts of this work are difficult to predict. However, as a deep model, our method shares some common pitfalls of the standard deep learning models, e.g., demand for powerful computing resources, and vulnerability to adversarial attacks.
- UCF $_{101}$ Unknown
- HMDB<sup>7</sup> . CC BY 4.0
- Jester<sup>8</sup> Unknown
- Epic-Kitchens  ${}^{9}$  CC BY-NC 4.0
- Sprites<sup>10</sup> CC-BY-SA-3.0
I3D11 Apache License 2.0
-  $\mathrm{TRN}^{12}$  BSD 2-Clause License
- CoMix13 . Apache License 2.0
-  $\mathrm{TA}^3\mathrm{N}^{14}$  MIT License
$\cdot \mathrm{CO}^{2}\mathrm{A}^{15}$  Unknown
- PyTorch<sup>16</sup> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
-  $\mathbf{Netron}^{17}$  MIT License
- flops-counter.pytorch<sup>18</sup> MIT License
$^{6}$ https://www.crcv.ucf.edu/data/UCF101.php  
 $^{7}$ https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database  
 $^{8}$ https://20bn.com/datasets/jester  
 $^{9}$ https://epic-kitchen.s.github.io/2021  
 $^{10}$ https://github.com/YingzhenLi/Sprites  
 $^{11}$ https://github.com/piergiaj/pytorch-i3d  
 $^{12}$ https://github.com/zhoubolei/TRN-pytorch  
 $^{13}$ https://github.com/CVIR/CoMix  
 $^{14}$ https://github.com/cmhungsteve/TA3N  
 $^{15}$ https://github.com/vturrisi/C02A  
 $^{16}$ https://github.com/pytorch/pytorch  
 $^{17}$ https://github.com/lutzroeder/netron  
 $^{18}$ https://github.com/sovrasov/flops-counter.pytorch