RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Figure 3: Our proposed model architecture, the Dual Encoder Transformer for Sign Language Generation. Our architecture is characterized by using two encoders, one for text and one for gloss annotation. The use of two encoders allows to multiply the outputs of both emphasizing the differences and similarities. In addition we to using skeleton poses and facial landmarks, we include facial action units Friesen and Ekman ( 1978 ) .; Our proposed model architecture, the Dual Encoder Transformer for Sign Language Generation. Our architecture is characterized by using two encoders, one for text and one for gloss annotation. The use of two encoders allows to multiply the outputs of both emphasizing the differences and similarities. In addition we to using skeleton poses and facial landmarks, we include facial action units Friesen and Ekman ( 1978 ) .
Figure 4: Comparison of the ground truth and the generated poses with our proposed dual encoder model for the gloss annotations cloud and rain . The upper example shows that the predictions captured the correct hand shape, orientation, and movement of the sign cloud . In the lower example it is visible that the predictions captured the repeating hand movement meaning rain . Although at first glance the hand orientation seems not correct, it is a slight variation which still is correct.; Comparison of the ground truth and the generated poses with our proposed dual encoder model for the gloss annotations cloud and rain . The upper example shows that the predictions captured the correct hand shape, orientation, and movement of the sign cloud . In the lower example it is visible that the predictions captured the repeating hand movement meaning rain . Although at first glance the hand orientation seems not correct, it is a slight variation which still is correct.
Figure 5: Examples in which our model failed to generate the correct phonology of signs. Example 1 depicts inaccuracies in hand shape, orientation, and movement. Example 2 shows the difficulty of the model to capture pointing hand shapes.; Examples in which our model failed to generate the correct phonology of signs. Example 1 depicts inaccuracies in hand shape, orientation, and movement. Example 2 shows the difficulty of the model to capture pointing hand shapes.
Occurrence of different Part-of-Speech (POS) in the sign gloss annotation and the German transcripts computed with Spacy Honnibal and Montani ( 2017 ) . Although gloss annotations show fewer samples for all POS, the difference in the occurrence of adjectives is statistically significant with p<0.05 .
Translation results of the SLT model Camgoz et al. ( 2020 ) used for backtranslation when trained and evaluated with ground truth hand and body skeleton joints (manual) and facial landmarks and AUs (facial).
Back translation results obtained from the generative models when using only manual features. Our proposed model has the highest scores in almost all metrics compared to the models using only gloss or text.
Back translation results obtained from the generative models when using manual features and facial landmarks and AUs. Our proposed model has the highest scores in all metrics compared to the models using only gloss or text.