# DanceCaps: Pseudo-Captioning for Dance Videos Using Large Language Models
## 3 Pseudo-Captioning Based Dance Caption Generation
### Tags
#### 3.1.1 Motion Tags  
Motion tags are tags that describe a dancer's movements over time, encompassing both the physical changes in body movement and the detailed linguistic descriptions of the dancer's actions throughout the sequence. These tags not only visually represent the progression of the dance over time but also capture the overall dynamics of the movements. The physical variation in body movement over time is tracked by monitoring the positions of a dancer's joints in video frames, which not only captures the overall characteristics of the body's motion but also indirectly reflects the tempo and mood of the entire dance. To achieve this, the joint positions of the dancer in each video frame must first be detected, and in this study, OpenPose [33] was used for joint position estimation. The positions of 14 joints, as shown in Figure 2, were extracted, and the speed and acceleration of each joint over time were calculated based on these position values. To capture the overall characteristics of the body's movement, the speed and acceleration values of all joints were averaged to derive the body's mean speed and acceleration. These values were then averaged over time to calculate the body's overall mean speed and acceleration. Rather than directly using the calculated average speed and acceleration, these values were translated into descriptive terms that could be intuitively utilized by the LLM to generate captions. The mean speed and acceleration values were categorized into five groups each and used as tags. For category selection, the average speed and acceleration of the dancer were computed for all single-dancer videos in the AIST++ dataset, and the 20 th, 40 th, 60 th, and 80 th percentiles of this distribution were chosen as thresholds. Based on these thresholds, speed was divided into five categories: rapid, high, moderate, low, and slow, while acceleration was categorized as rapid, high, moderate, gradual, or slight. \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline
**Category** & **Tag Name** & **Example** \\ \hline \multirow{8}{*}{**Motion Tags**} & \multirow{8}{*}{Temporal motion} & The dancer begins in a waacking pose, then changes the movement by stretching out her right arm and pulling her right arm inward while lifting her left arm in a cross motion. \\ \cline{2-3}  & & Average velocity of movement & Slow, Low, Moderate, High, Rapid \\ \cline{2-3}  & & Average acceleration of movement & Slight, Gradual, Moderate, High, Rapid \\ \hline \multirow{2}{*}{**Dance Tags**} & Dance genre & “Hip-hop”, “Break”, “Krump” \\ \cline{2-3}  & & Choreography step & “Walking out”, “whip”, “Brooklyn step” \\ \hline \multirow{8}{*}{**Music Tags**} & Music genre & “Hip-hop”, “Electronic”, “Jazz” \\ \cline{2-3}  & Music tempo & Low, Medium, High tempo \\ \cline{2-3}  & Beat location & [1.22, 4.55, 7.88,...], frame position \\ \cline{2-3}  & & \multirow{2}{*}{Instrument} & “Piano”, “Synthesizer”, “Acoustic guitar” \\ \cline{2-3}  & & Average pitch level & Low, Medium, High \\ \cline{2-3}  & & Music mood & “Happy”, “Energetic”, “Film” \\ \hline \multirow{2}{*}{**Other Tags**} & Dancer characteristics & “Man in a black shirt and gray pants” \\ \cline{2-3}  & Spatial characteristics & “Courtyard” \\ \hline \end{tabular}
\end{table} 
Table 2: The types of tags and their corresponding examples. In addition to the physical calculation of body movement, temporal motion tags were extracted to describe the dancer's detailed movements in text. To achieve this, video captioning algorithms were utilized. However, the algorithms proposed by [PERSON] et al. and [PERSON] et al. (2018); [PERSON] et al. (2018); [PERSON] et al. (2018); [PERSON] et al. (2018) focus more on describing the task depicted in the video rather than the temporal details of the movement.