Learning Adverbs with Spectral Mixture Kernels♰♰♰Tomoe Taniguchi Daichi Mochihashi♰
Ichiro Kobayashi♰ Ochanomizu University ♰♰The Institute of Statistical Mathematics

1. Introduction

4. HDP-Spectral Mixture LDABackground• Technological advancements are making household robots that assist in daily tasks a reality• Effective human-robot collaboration requires sharing and understanding experiences
through language

Algorithm

Overview
Objective :We focus on human actions to understand the meanings of adverbs through motion
features
Dimensionality Reduction:
We use Gaussian processes to compress human motion data and extract frequency
information
Joint Topic Model:
We propose a joint topic model which learns the relationship between human motions andadverbs to understand the meanings of adverbs related to human actions

2. Human Motion Representation• Human motion can be represented
as smooth trajectories• We use Gaussian Process Latent
Variable Model(GPLVM)[Lawrence, 2003] to describe thehuman motions
Figure 1 : Motion data compressed by GPLVM・ Three walking trajectories processed through GPLVM visualized in the three-dimensional latent space
・ Cyclicity of the representations reflects the periodicity of human movements

3. Frequency components in a motion

Figure 3 : Graphical model of HDP-SMLDA• We employ collapsed Gibbs sampling [Griffiths and Steyvers, 2004] as the learningalgorithm for estimating the topic distribution of adverbs and frequencies in the HDPSMLDA• We estimate the number of topics (K) from the data using the Chinese Restaurant Process
Here,
G0 : base distribution
D : The number of videos
K : The number of topics
Q : Dimensionality of frequencies
N : The number of adverbs
M : The number of frequencies
Θ : Topic distribution
Z : The latent variables of adverbs
W : Adverbs
Φ : Word distribution
η : The parameter of φ
Y : The latent variables of frequencies
X : frequencies
μ : Mean of Gaussian distribution
Σ (= σ2) : Variance of Gaussian distribution

Sampling topics of adverbs• η is iteratively updated using the FixedPoint Iteration method• Σ is learned as a fixed value• µ is sampled from the gaussian
distribution (λ=1/σ2)

Sampling topics of frequencies

SM kernel enables automatic learning of a mixed kernel from data by considering a combined Gaussian
distribution in the Fourier domain
basis function• Human motion is cyclical• We use Spectral Mixture kernel(SM kernel)[Wilson and Adams,2013] to extract frequency
components from human motions
Figure 2 : Frequency information extracted from thecompressed data

・ We analyzed the motions depicted in Figure 1 using the Spectral Mixture kernel
・ The vertical and horizontal axes respectively represent the probability density and mean of theestimated four Gaussian distributions

5. Dataset

Preprocessing of Videos

100 Walks (Walk data)• Walk video in 2D format on Youtube• Required 3D pose information for the experiment• Divided video into 100 segments at motion breaks• Applied four methods for 3D pose estimation

6. Experiment

AIST++ (Dance data)• Curated dance videos with copyright-cleared music• Created and maintained by AIST
• Annotations in COCO format for 16 joint points [Li et al. ,2021]• We requested Japanese adverb annotations for each video using the crowdsourcing
site Lancers• We utilized the direction vectors connecting each joint as input data
• To account for individual differences such as arm length, we compute unit vectors
Table 1 : Details of input data

Results

Generation of Adverbs from frequencies

Experimental Settings• Using synonyms or adverbs labeled in other video data, appropriate adverbs can be inferred• A Q value of 10 results in more accurate adverb predictions.

Input:
Adverb data, Frequency data

Table 3 : Perplexity of topic models

Datasets:
AIST++ 1,063 videos
Optimize Parameters:
we optimize
Concentration parameter α
Word distribution parameter ηFrequency distribution parameter m,βEpochs: 1,000Figure 4 : Distribution of frequencycomponents categorized by topic (AIST++)• The learned Gaussian distribution means (μ) arescattered for each topic• Topics with similar μ distances indicate similaractions• Frequency information contributes to theclassification of adverbial topics

Evaluation: Perplexity of words

Figure 5 : A video for evaluation

Table 3 : Top 7 adverbs estimated by HDP-SMLDA

Comparative Results•••

High scores, which does not necessarily indicate effective learning of adverbs
Our model demonstrated the lowest scores on both datasets
Our model showcases the ability to accurately estimate adverbs even with limited data

Frequency sampling:
Using the weights from SM Kernelestimation, sample frequencies foreach epoch

Experimental Settings

Table 2 : Top 5 adverbs in each topic estimated by HDP-SMLDA (AIST++)

Table 4 : Perplexity of NN models••••

Hidden layer size: 128
Optimization function: SGD
Loss function: Cross-entropy
Number of epochs: 1000

7. Conclusion• We have proposed HDP-SMLDA, which aims to comprehend the semantic nuances of sensory adverbs pertaining to human motions by learning cooccurrence relationships between motion features and adverbs.
• When compared to the other representative models, our model exhibits superior performance on classification of adverbs.