# DanceCaps: Pseudo-Captioning for Dance Videos Using Large Language Models
## 1 Introduction  
Dance encompasses both visual information from movements and auditory information from music, making it essential to consider these multimodal characteristics in dance research. With recent advancements in deep learning, the dance field has rapidly evolved. In particular, dance generation research has significantly progressed through the use of deep learning models such as RNNs [1], transformers [2], and diffusion [3] models, leading to the creation of more sophisticated dance movements. Despite these advancements, there remains a lack of comprehensive datasets containing the necessary information for training and evaluating these models. As in other fields, dance research requires not only video data, but also datasets that include tags or captions that capture detailed information about the dance itself. In contrast, vision language models utilize publicly available image-label paired datasets such as MS-COCO Captions [4], Flickr30k Captions [5], VQA 2.0 [6], ImageNet [7], and LAION-5B [8] to address tasks like image captioning, text-to-image generation, and visual question answering. Additionally, CLIP [9] and BLIP [10] are multimodal models that simultaneously understand images and text, having been trained on large-scale web data consisting of image-text pairs. Similarly, audio language models rely on audio-label paired datasets like AudioCaps [11], Clotho [12], and MACS [13] for tasks including audio captioning, text-to-audio generation, and audio question answering. Both vision and audio research benefit from the availability of open or private datasets thatfacilitate ongoing model training and evaluation through benchmarking. Dance research, however, has lagged behind these traditional fields, with fewer open datasets and limited label information. The most commonly used open dance datasets in dance research currently include the AIST dance dataset [14], AIST++ dance dataset [15], and AniDance dance dataset [16], with the AIST++ dance dataset being the most widely utilized. However, these datasets mainly focus on the joint coordinates of dance motions and provide limited labeling, so more specific research requires the creation of research-specific datasets. Examples of such individual studies of datasets include [PERSON] et al. [17], which used the game 'Just Dance' as a dataset; [PERSON] et al. [18] and [PERSON] et al. [19], which utilized YouTube videos of specific dance genres; and [PERSON] et al. [20], which utilized regional music features. These private datasets often lack transparency about their composition, quality, and features, making it difficult to analyze the models trained on them and the results obtained. Furthermore, the lack of datasets focused on dance-related labels, tags, or dance captions makes it more difficult to conduct research related to dance labels. Therefore, the development of a dance caption dataset with paired dance movements and natural language descriptions could significantly expand research possibilities. Unlike previous methods requiring direct annotation of dance-caption pairs, recent advancements in LLM and pseudo-captioning techniques present new opportunities for generating such paired datasets. These technological advancements are expected to facilitate the creation of dance caption datasets, enhancing the scope and depth of choreographic research. This paper focuses on generating captions for dance videos that lack existing labels or tags by leveraging multiple information sources. We propose a pseudo-captioning method that generates captions by selecting meaningful tags according to the goal. First, the visual and audio features extracted from the dance videos are classified into four groups: movement, dance, music, and others, and each category is subdivided into detailed tags and extracted. Then, we propose a method to generate three types of captions using LLMs by utilizing the extracted tags and instructions suitable for the purpose of each caption.