Learning Adverbs with Spectral Mixture Kernels (2024)

Abstract: For humans and robots to collaborate more in the real world, robots need to understand human intentions from the different manner of their behaviors. In this study, we focus on the meaning of adverbs which describe human motions. We propose a topic model, Hierarchical Dirichlet Process-Spectral Mixture Latent Dirichlet Allocation, which concurrently learns the relationship between human motions and adverbs by capturing the frequency kernels that represent motion characteristics and the shared topics of adverbs to depict such motions. We trained the model on datasets we made from movies about “walking” and “dancing”, and found that our model outperforms representative neural network models in terms of perplexity score. We also demonstrate our model’s ability to estimate suitable adverbs for a given motion automatically extracted from a movie.

heading: 1 Introduction
text: With technological innovations in artificial intelligence, the widespread use of household robots that collaborate with humans to assist them in their daily lives is becoming a reality. In order for these robots to collaborate with humans, it is important to share and understand their experiences through language, because language is the most convenient communication tool capable of conveying human experience and knowledge. With this background, research on language use by robots in the real world has been actively studied (Taniguchi et al., 2019; Tellex et al., 2020; Kalinowska et al., 2023; Karamcheti et al., 2023). Significantly, within this domain, Large-scale language models (LLMs) such as OpenAI’s ChatGPT1 and Google’s PaLM (Chowdhery et al., 2022) are also used to control robots. ChatGPT is used to execute various types of robotics tasks (Vemprala et al., 2023), and PaLM-SayCan (Ahn et al., 2022) and
1https://chat.openai.com.
PALM-E (Driess et al., 2023) have been developed based on PaLM (Chowdhery et al., 2022). Singh et al. (2023) and Huang et al. (2022) have proposed methodologies for generating task plans for robots that employ LLM. Their approach conveys robot’s motion plans through a chain-of-thought framework (Wei et al., 2022). Though they are good at describing the general plan of action of a robot in language to accomplish a specific task, the language description does not capture the precise correspondence between nuanced expressions and the actual robot behaviors in the real world. Furthermore, the focus of their studies is not on the verbal representation of the behaviors of the observed object by a robot, but on the robot’s action plan. For the advancement of robotics, it becomes imperative to comprehensively and statistically grasp the repertoire of “motions” that humans genuinely exhibit, as well as discern the variations in individual characteristics and contextual nuances associated with them. These insights should be aptly assimilated within the robotic systems. Building upon the aforementioned, we shall address this challenge by casting our focus on adverbs to mathematically establish a correspondence between motions and adverbs that represent them.

heading: 2 Related work
text: Research is being conducted to elucidate the relationship between motions and the natural language that describes them. Bidirectional conversion models from natural language descriptions to motions, or vice versa, using sequence-to-sequence (Seq2seq) (Sutskever et al., 2014) learning have been proposed by Yamada et al. (2018); Plappert et al. (2018); Ito et al. (2022). Though these models can achieve bidirectional conversion between language and motion sequences, it lacks in learning the correspondence between the manner of motions and the language that represent them. Furthermore, in the conventional research the fo-
7742
cus has predominantly revolved around finite motions, such as “take” and “put”, which were preconceived by humans, thereby neglecting the pursuit of methodologies that facilitate the adaptable modulation of multiple motions contingent upon contextual cues. In this paper, we focus on capturing the relationship between verbs and adverbs to flexibly express actions using the function of adverbs. Limited research has been conducted thus far to delve into the semantic comprehension of adverbs. Notable instances within this domain include the Three-Stream Hybrid Model (Pang et al., 2018), which employs Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) and inceptionV3 (Szegedy et al., 2016) to acquire knowledge related to adverbs. Additionally, Action Modifiers (Doughty et al., 2020), which employ an I3D network (Carreira and Zisserman, 2017) and scaled dot-product attention (Vaswani et al., 2017) to discern the impact of adverbs on motion sequences. These models employ image features derived from videos, such as RGB and optical flow (Simonyan and Zisserman, 2014), as representations of motions. However, these representations fail to capture the intrinsic essence of the motions themselves because they are unable to discern the component of motions denoted by the adverb. Therefore, unlike conventional research approaches, in this study, we focus on the frequency components that make up human motion and attempt to express the motion by those components. By doing so, we aim to enable the robot to understand the meaning of adverbs related to motions such as “cut roughly”, “dance dynamically”, and so on.

heading: 3 Joint Topic Model of Motions and Adverbs
text: For this purpose, we propose a new topic model, Hierarchical Dirichlet Process-Spectral Mixture Latent Dirichlet Allocation (HDP-SMLDA) to capture the relationship between the frequency components of human motions and the adverbs that describe them. The proposed model makes it possible to establish a statistical correspondence between adverbs and nuances associated with motions. This enables the control of robot actions through verbal instructions, such as “handle with more caution” or “cut roughly”, and it is also possible to make the robot understand human intentions due to slightly different manner of movement. On the contrary, from the perspective of natural language process-
ing, it has been impossible to express the actual meaning behind words like “freely” or “flexibly”. However, the integration with robotics makes it possible for the first time to represent their meaning, allowing not only the description of actions through language but also the generation of actions from language cues.

heading: 3.1 Human Motion Representation
text: Since human motion is represented as a smooth trajectory, we use a Gaussian process (GP) (Rasmussen and Williams, 2006), which is defined as a distribution over functions, to describe the motions. In a GP, the kernel function k(x, x′) that determines the similarity between two data points (x, x′) is applied to the data set to compute the covariance matrix and estimate the predictive distribution. The choice of kernel function is an important factor that affects the behavior and performance of the GP model. GP models are primarily used for regression and classification, fundamental techniques that are also widely used by the natural language processing community (Cohn et al., 2014). We utilize Gaussian Process Latent Variable Model (GPLVM, see Appendix A) (Lawrence, 2003), when nonlinearly compressing high-dimensional human motion data into low-dimensional trajectories (Section 4.1 gives the details). In Figure 1, we show three walking trajectories processed through GPLVM visualized in the three-dimensional latent space. Cyclicity of the representations reflects the periodicity of human movements.

heading: 3.2 Frequency components in a motion
text: Wilson et al. (Wilson and Adams, 2013) introduced a technique known as the Spectral Mixture kernel (SM kernel), which enables automatic learning of a mixed kernel from data by considering a combined Gaussian distribution in the Fourier domain.
This approach surpasses the limitation of utilizing pre-existing kernels or their combinations in Gaussian processes. As a fundamental component of the Gaussian process, we consider a radial basis function k(τ) that solely depends on τ = x − x′. According to Bochner’s theorem (Bochner et al., 1959; Stein, 1999), any k(τ) can be expressed in the following equation:
k(x, x′) = k(τ) = ∫
R e2πis Tτψ(s)ds. (1)
As k(τ) is considered equivalent to probability density ψ(s) in the frequency domain, we consider a mixture of Gaussian distributions for ψ(s). Each component of the Gaussian distributions is equivalent to considering the following basis function in the original domain:
k(τ |σ, µ) = exp(−2π2τ2v2) cos (2πτµ). (2)
Thus, we can consider a mixture of M basis functions for a latent kernel. Here, µqm and v q m represent the mean and variance, respectively, of the q-th dimension of the input X in the m-th basis:
k(τ)= M∑
m=1
wm cos (2πτ Tµm)
Q∏
q=1 exp(−2π2τ2q vqm). (3)
The weights parameter w, mean µ, and variance v can be learned through hyperparameter optimization of Gaussian processes. As shown in Figure 2, we employ this method to extract M frequency components (represented by the mean µ) that are expected to be relevant to adverbs from the Qdimensional latent trajectories X shown in Figure 1. These components are then used as observed values that capture the characteristics of the motions. It
is worth noting that while the trajectory in X can be directly Fourier transformed, doing so would not allow us to distinguish between the function passing through particular points (the phase of the function) and the characteristics of the function itself.

heading: 3.3 Hierarchical Dirichlet Process-Spectral Mixture LDA
text: The extracted frequency components from the motions are assumed to be associated with the adverbs assigned to those motions. By leveraging a Gaussian-Multinomial LDA (GM-LDA) (Blei and Jordan, 2003), we can cluster the frequency components and adverbs simultaneously into topics, thereby identifying frequency components that are likely to co-occur with a given adverb. It is important to note that GM-LDA required the number of topics K to be known in advance. However, the number of topics is typically unknown, and assuming prior knowledge of this parameter is a significant limitation. To address this issue, we propose the Hierarchical Dirichlet Process Spectral Mixture LDA (HDP-SMLDA), which automatically estimates the number of topics from the data by incorporating a hierarchical Dirichlet process (Teh et al., 2006) into GM-LDA. The graphical model, as depicted in Figure 3, considers Q as the number of dimensions of the frequency components. In our study, we set Q = 3 because we preprocessed the data by GPLVM into three-dimensional latent space. The number of kernel mixtures M in the Spectral Mixture (SM) kernel discussed in the previous section is denoted as Md in this model. Adverbs are sampled from a multinomial distribution, while the frequency component is treated as continuous data emitted from a Gaussian distribution associated with each latent topic. Let us assume the existence of a potential topic distribution θd for each motion d. The dimensionality of the topics, denoted as K, is variable, allowing for
flexibility. The generative process of the adverb wdn (n = 1, . . . , Nd) and the frequency component xdm (d = 1, . . . , D; m = 1, . . . ,Md), which refers to the µ in Equation 3, associated with the motions is outlined as follows:
1. Draw G0 ∼ DP(γ,H). 2. For d = 1 . . . D, – Draw θd ∼ DP(α,G0). 3. For n = 1 . . . Nd,
– Draw zdn ∼ θd – Draw wdn ∼ ϕzdn .
4. For m = 1 . . .Md, – Draw ydm ∼ θd – Draw xdm ∼ N (µydm , σ2ydm).
In the generative process, ϕk represents a multinomial distribution over adverbs that corresponds to the k-th topic, while N (µk, σ2k) denotes the Gaussian distribution for the observed frequencies associated with that topic. The topic distribution θ is computed based on the information from both the adverbs and frequency components. This topic distribution is then utilized to assign topics to each adverb and frequency component iteratively for each motion d.
Sampling Topics of Adverbs and Frequencies
We employ collapsed Gibbs sampling (Griffiths and Steyvers, 2004) as the learning algorithm for estimating the topic distribution of adverbs and frequencies in the HDP-SMLDA.
Sampling topics of adverbs Let T represents the set of table assignments and ℓ denotes the table number. According to the Chinese restaurant process (Teh et al., 2006), the topic zdn assigned to the adverbwdn is determined by sampling the occupied table Tdn using the following formula. Here, ℓused and ℓnew correspond to existing and new tables, Lk and L represent the number of tables assigned to topic k and the total number of tables, respectively, and V is the number of vocabularies (“\” denotes exclusion of that index):
p(tdn = ℓ|W,T\dn,Z,Y, α, γ, η)
∝ { p(tdn = ℓused)|W,T\dn,Z,Y, α, γ, η) p(tdn = ℓnew)|W,T\dn,Z,Y, α, γ, η)
∝    (Ndl\dn + ∑Q q=1M q dl) Nkwdn\dn + η Nk\dn + ηV ∑K
k=1 αLk L+γ
Nkwdn\dn + η
Nk\dn + ηV +
αγ
L+ γ
1 V .
(4)
The following formula is employed to sample the topics assigned to the new table. Here, kused refers to existing topics, while knew represents new topics:
p(zdl = k|W\dn,T,Z\dl, α, γ, β)
∝ { p(zdl = kused|W\dn,T,Z\dl, α, γ, β) p(zdl = knew|W\dn,T,Z\dl, α, γ, β)
∝    Lk · Nkwdn + η Nk\dn + ηV
γ · 1 V
. (5)
The hyperparameter η is iteratively updated using the Fixed-Point Iteration method (Minka, 2003) based on the following equation, where Ψ(x) = d/dx log Γ(x):
η′ = η · ∑K k=1 ∑V v=1Ψ(Nkv + η)−KVΨ(η)
V ∑K k=1Ψ(Nk + ηV )−KVΨ(ηV ) .
(6)
Sampling topics of frequencies The topic ydm assigned to the frequency component xdm is sampled using the following equations:
p(tdm = ℓ|W,T\dm,Z,Y, α, γ, η)
∝ { p(tdm = ℓused|W,T\dm,Z,Y, α, γ, η) p(tdm = ℓnew|W,T\dm,Z,Y, α, γ, η)
∝    (Ndl + ∑Q q=1M q dl\dm)N (x|µk, σ2k) ∑K k=1 αLk L+ γ N (x|µk, σ2k) + αγ
L+ γ N (x|µknew , σ2knew),
(7)
p(zdl = k|X\dm,T,Y\dl, α, γ, β)
∝ { p(zdl = kused|X\dm,T,Y\dl, α, γ, β) p(zdl = knew|X\dm,T,Y\dl, α, γ, β) ∝ { Lk · N (x|µk, σ2k) γ · N (x|µknew , σ2knew) . (8)
The variance parameter σ2 of the Gaussian distribution is learned as a fixed value. To ensure that the Gaussian distribution is evenly distributed over the data range, we calculate σ using the following equation. This is done because the data typically fall within the range of approximately −3σ to 3σ when the mean is set to 0. Here, K+ represents the number of topics at the current iteration:
σq = max(Xq)−min(Xq)
6K+ . (9)
The mean parameter µ of the Gaussian distribution is sampled from the posterior distribution given
by the following equation. Here, λ is defined as λ = 1/σ2, where σ2 represents the variance of the Gaussian distribution:
p(µ|Y) = N (µ|m, (βλ)−1) . (10) Let us assume that β0 and m0 are the parameters of the prior distribution, and they are defined as follows:
β =M + β0, m = 1
β
( M∑
m=1
xm + β0m0
) .
(11)
To estimate the mean µknew for the Gaussian distribution associated with the new topic directly is not possible since there is no data belonging to the cluster. To address this, the mean is sampled from a Gaussian distribution using suitable parameters, allowing it to be learned to some extent, and then estimated as same as the mean of existing topic.
Estimation of scaling parameter α To better estimate the number of topics that best fit the data, we adopt a gamma distribution as the prior distribution for the scaling parameter α:
p(α|π, s, Z, c1, c2) = Gamma (α|c1+K+−s, c2−log π).
(12)
π and s are sampled as follows:
p(π|α, s, Z, c1, c2) = Beta (π|α+ 1, N +M),
p(s|α, π, Z, c1, c2)
= Bernoulli ( s ∣∣∣∣ N +M
N +M + α
) . (13)

heading: 4 Experiments
text: We begin by providing a description of the datasets utilized in our experiments. We then proceed to conduct an experiment involving HDP-SMLDA, where we examine the adverbs and frequency components, and generate adverbs based on the frequency components within the trained model.

heading: 4.1 Experimental settings
text: Datasets We conducted an experiment utilizing a dataset containing walking motions called 100 Walks2and another dataset comprising dancing motions called AIST++3.
100 Walks 100 Walks, the video available on YouTube, is in a two-dimensional format. However, for our experiment, we required three-dimensional pose information as input data. To overcome this limitation, we divided the video into 100 segments at the motion breaks and applied four different methods for three-dimensional pose estimation.
1. Estimate 2D skeletal coordinates from video data using Openpose (Cao et al., 2021) (Figure 4(a)) 2. Estimate the depth of the video per frame using FCRN-depth prediction (Laina et al., 2016) (Figure 4(b)) 3. Estimate 3D skeletal coordinates from video data using results of 1 and 2 ,and 3d-pose baseline (Martinez et al., 2017) (Figure 4(c)) 4. Normalize human body orientation using a rotation matrix (Figure 4(d))
AIST++ The AIST Dance DB (Tsuchida et al., 2019) is a curated dataset consisting of original dance videos. These videos have been carefully selected and include dance performances accompanied by copyright-cleared music. The dataset is created and maintained by the National Institute of Advanced Industrial Science and Technology (AIST). Li et al. (2021) conducted annotations on the AIST Dance DB dataset, specifically focusing on three-dimensional human keypoints and developed a dance generation model. These annotations provide valuable information for each dance video
2https://www.youtube.com/watch?v= HEoUhlesN9E
3https://google.github.io/ aistplusplus_dataset/
in the dataset. Additionally, they released the annotated dataset called AIST++, which consists of 1,199 simple Basic Dance motions annotated with three-dimensional pose information for 16 joint points in the COCO format. The dataset consists of 10 different choreographies, each representing a specific genre of dance. For each choreography, there are 20 different dancers who perform the dance in the corresponding video. The dancers follow the specified choreography while dancing to genre-specific music. The music tempo varies across the dataset and is set at six different levels.
Preprocessing of Videos
We employed a crowdsourcing system called Lancers4 to get annotations from multiple annotators for the Japanese adverbs associated with the human motions in the videos. Appendix B describes more details of our crowdsourced experiments. In comparison to data set used in prior research (Pang et al., 2018; Malmaud et al., 2015), we have amassed a more extensive corpus of adverbs in both datasets. In addition, we utilize the direction vectors connecting each joint as input data . To account for individual differences such as arm length, we compute unit vectors (see Appendix C for details).
4https://www.lancers.jp/
Extraction of Frequency Components from Human Motions
Frequency components were extracted from the preprocessed video data utilizing the following two steps. Experiments were conducted by varying the number of kernel mixtures, denoted as Md, within the range of 4 to 12.
1. Reduce high-dimensional pose data to lowdimensional latent variables using GPLVM. Figure 1 shows the case of reducing pose data into three-dimensional latent variables. 2. Extract frequency components for each dimension from the three-dimensional latent variables using SM kernel. Figure 2 shows the case of using four bases of Gaussian distribution (see Appendix D for details).
The SM kernel is optimized with weights as parameters, representing the significance of each frequency component. At each learning iteration of HDP-SMLDA, the frequency components used as motion features in each video are sampled using the weights.

heading: 4.2 Results
text: For the AIST++ dataset with Md = 4 and learning epochs set to 1,000, Table 1 displays the top five words for each adverb, listed the Normalized Pointwise Mutual Information (NPMI) values (Bouma, 2009) for each adverb in each topic in descending order from the learned topic-word distribution. We confirmed that the model’s perplexity has converged after training. In typical topic models, words with high co-occurrence tend to be placed in the same topic (Appendix E). However, HDPSMLDA incorporates not only words but also behavioral information into clustering. Consequently,
if actions are similar, there is a possibility that words indicating opposite meanings, as observed in Topic 6 (joyfully and uninterestedly), may be placed in the same topic. Figure 5 visually represents the 100 samples in a three-dimensional space, obtained from the Gaussian distribution associated with the mean µk of each learned topic. Each sample represents a frequency component that symbolizes a specific topic, and the proximity of the samples indicates similarity in their frequency components. It is important to note that since the scales are not estimated, the dispersion of the points in the figure remains constant. To evaluate the performance of this model, perplexity is used as a metric. Table 2 presents the perplexity of each topic model during training. Additionally, the perplexity for the Unigram model is calculated using the word distribution prior to training.

heading: 4.3 Discussions
text: Generation of Adverbs from Frequency To verify the accurate association between frequencies and adverbs, we performed an experiment where we generated adverbs based on the frequency components extracted from an evaluation video (Figure 6), utilizing the learned word distribution. Table 3 presents both the ground truth adverbs and the top seven adverbs with the highest probabilities, calculated through HDP-SMLDA. Through the estimation of Md from 4 to 12, we observed that, for the majority of evaluation videos, the estimation withMd = 10 yielded more suitable adverbs as the top choices.
In Figure 5, the arrangement of the 16 Gaussian distributions evenly spans the width of the data. Notably, Topic 5 and Topic 14 exhibit prox-
imity to each other, indicating a similarity in the content of the motions, as supported by Table 3 showcasing the top adverbs associated with each topic. Topics 1, 8, and 10 appear more distanced from the other topics. Notably, these three topics demonstrate pronounced adverb features in terms of frequency. While there may be an apparent overlap between the content of Topics 1 and 10, a closer examination of the top 20 words reveals that Topic 1 encompasses emotionally driven dances such as “bravely” and “heavily”, while Topic 10 represents adverbs associated with more vigorous movements like “sharply” and “refreshed”. This distinction suggests that the model successfully clusters adverbs based on both semantic and motion-related features derived from frequency components. It is also a reasonable result that words with opposite meanings in language are assigned to the same topic such as “joyfully” and “uninterestedly” in Topic 6 due to the similarity of motions. The perplexity values from Table 2 indicate significantly lower values compared to those obtained from LDA training data, signifying the valuable contribution of frequency components in adverb topic classification. Although increasing the number of mixtures in the kernel was expected to reduce perplexity, the experiment yielded unfavorable results. On the other hand,regarding the generation of adverbs from frequency components, it was observed that when Md = 10, the model was able to estimate more suitable adverbs compared to when Md = 4. This observation raises the possibility that the an-
notators may have encountered difficulty in identifying the precise vocabulary during the annotation process or that the model could generate correct synonyms that did not align perfectly with the ground truth.
Comparative Results We conducted additional experiments to compare our method with RedWine (Misra et al., 2017) and AttributeOp (Nagarajan and Grauman, 2018). Due to significant discrepancies in the data structures of the input, we did not use prior studies in the comparative experimentation. Given that our study involves annotations of multiple adverbs per video, multi-label learning becomes necessary. These models are designed to learn attributes of objects in images. We adapt by substituting actions for objects and adverbs for attributes. We extracted features from the videos by using the pre-trained S3D model (Miech et al., 2020) from HowTo100M (Miech et al., 2019), which served as inputs to the model. In typical class classification learning, the model calculates the error by back-propagating the difference between the output probability and the input label. However, in our case, training is performed by back-propagating the average of errors for all adverb labels annotated to the video. Table 4 displays the perplexity scores for each model during evaluation.Our method outperforms all baselines. From these results, it is evident that for tasks involving the classification of adverbs related to human motion, capturing motion in terms of frequencies is more effective than using image features such as RGB or optical flow. We also conducted experiments using Long Short-Term Memory (LSTM) and Multi-Layer Perceptron (MLP)(Rumelhart and McClelland, 1987), with four different data inputs:
1. Input data processed by GPLVM to LSTM 2. Input original data to LSTM 3. Input frequency (Md = 4) to MLP 4. Input frequency (Md = 10) to MLP
We conducted experiments by configuring both models with a hidden layer size of 128, utilizing SGD as the optimization function, and employing cross-entropy as the loss function. Furthermore, the number of epochs was set to 1000 to align with the experiments in our model. Table 4 displays the perplexity scores for each model during evaluation. Comparing the data processed by GPLVM with the original data, it is evident that the processed data yielded lower perplexity, indicating the effectiveness of data dimensionality reduction in class
classification. All neural network models received high scores, which does not necessarily indicate effective learning of adverbs. Nonetheless, our proposed method demonstrated the highest scores on both datasets, highlighting its superior performance. Thus,our model showcases the ability to accurately estimate adverbs even with limited data.

heading: 5 Conclusion
text: We have proposed a joint topic model named HDPSMLDA, which aims to comprehend the semantic nuances of sensory adverbs pertaining to human motions by learning co-occurrence relationships between motion features and adverbs. Within our framework, adverbs are modeled as a composite distribution within the frequency space of their kernels in a Gaussian process that represents the latent trajectory of motions. Consequently, it becomes feasible to estimate the constituents of sensory adverbial motions. When compared to the simple neural network model, our model exhibits superior performance on classification of adverbs. Our approach considers motions as a mixture of diverse frequency components, leading to the successful generation of appropriate adverbs from motion features in our empirical investigations.

heading: 6 Limitations
text: The primary limitation to the generalization of these results lies in the scarcity of datasets containing adverbially annotated human motions. There is no other way to annotate adverbs by ourselves to capture the meaning of adverbs which describe human motions, and it is difficult to make comparisons with other models because there are few studies working on the same research topic. Another limitation is that even if the adverbs output by the model are correct, such as synonyms, the model may judge that it has output the wrong one unless it is an exact match. We think this can be resolved by representing the adverbs in embedding vectors to evaluate output.

heading: 7 Ethical considerations
text: All datasets used in the experiments are either publicly available or have been licensed for use by the authors. In addition, all copyrights to the data generated using crowdsourcing were transferred to the authors.

heading: A Gaussian Process Latent Variable Model(GPLVM)
text: GPLVM is a probabilistic dimensionality reduction technique. It aims to find a low-dimensional representation of high-dimensional data while preserving the underlying structure of the data. This model assumes that the high-dimensional data is
generated by a non-linear transformation of a lowerdimensional latent space, where the latent variables follow a Gaussian distribution. By inferring the latent variables and the mapping from the latent space to the observed data, GPLVM can effectively capture the intrinsic structure of the data in a lowerdimensional space. It is widely used in various machine learning tasks such as data visualization, feature extraction, and clustering.

heading: B Additional Information for the annotation of adverbs
text: We requested each annotator to provide as many Japanese adverbs as possible for human motions of each video. To ensure the quality of the annotations, we considered only those adverbs that appeared at least three times across all the videos and discarded the rest as noise. For the 100 Walks dataset, we assigned 20 annotators to annotate every 100 videos. In the case of the AIST++ dataset, we assigned 5 annotators to annotate every 50 videos. This approach allowed us to collect a diverse range of adverbs associated with the motions while maintaining the quality of the annotations. The details of the adverb dataset are presented in Table 6, where the 100 Walks dataset is referred to as “walk” and the AIST++ dataset is referred to as “dance”. The metric “average adverbs” represents the mean number of adverbs annotated per video.

heading: C Details of preprocessing
text: For the 100 Walks dataset and AIST++ dataset, we compute 16 and 14 direction vectors, respectively.
The resulting vectors are then combined, with their three-dimensional coordinates arranged in the column direction for each frame. Consequently, the data dimensions are 48 and 42 for the respective datasets. We did this preprocessing for the reconstruction of the original pose information in the future.

heading: D Detailed description of Figure 2
text: In our approach, we employ the radial basis function (RBF) as the kernel function of GPLVM. To optimize the values of X and the hyperparameters of the kernel, we utilize the L-BFGS method (Liu and Nocedal, 1989). For Md = 4, the Gaussian distribution is depicted in Figure 2 with optimized mean µ and variance σ parameters for the first dimension of each motion, using the SM kernel. The estimated variance is exceptionally small, resulting in the Gaussian distribution being represented as a delta function in the figure. From Equation (3), we observe that a larger mean µ value corresponds to a shorter period. Therefore, it can be inferred that the spectral components representing the basis are more likely to be found on the left side of the spectrum for slow-moving motion data. Thus, (a) contains more fast motion components, (c) contains more slow motion components, and (b) lies in between as an intermediate case.

heading: E Adverbs in each topic by LDA
text: The results of clustering the AIST++ dataset using LDA are displayed in Table 5.In the table, the top five words in each topic are listed in descending order of their NPMI. The number of topics was set to 16 to align with the number of topics estimated by HDP-SMLDA.

heading: Future Work
text: We did this preprocessing for the reconstruction of the original pose information in the future