Multi-Instance Multi-Label Learning for Text-motion Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-14 03:58:49

Abstract: Text-motion retrieval (TMR) is a significant cross-modal task, aiming at retrieving semantically similar motion sequences with the given query text. Existing studies primarily focus on representing and aligning the text and motion sequence with the single embeddings. However, in the real-world, the motion sequence usually consists of multiple atomic motions with complicated semantics. This simple approach may hardly capture the complex relation and abundant semantics in the text and motion sequence. In addition, most atomic motions may co-occur and be coupled together, which further brings considerable challenges in modeling and aligning query and motion sequences. In this paper, we regard TMR as a multi-instance multi-label learning (MIML) problem, where the motion sequence is viewed as a bag of atomic motion and the text is the bag of corresponding phrase description. To address the MIML problem, we propose a novel multi-granularity semantics interaction (MGSI) approach to capture and align the semantics of text and motion sequences at various scales. Specifically, the MGSI first decomposes the query and motion sequences into three levels of event (bag), actions (instances), and entities, respectively. After that, we adopt graph neural networks (GNNs) to explicitly model their semantic correlation and perform semantic interaction at corresponding scales to align text and motion. In addition, we introduce a co-occurred motion mining approach that adopts the semantic consistency between the atomic motions as measurement to identify the co-occurred atomic motions.
These co-occurred atomic motions are fused and interacted with corresponding text to achieve a precise cross-modal alignment. We evaluate our methods on the HumanML3D and KIT-ML datasets, in which we achieve improvements in Rsum of 23.09\% on HumanML3D and 21.84\% on KIT-ML.

Main paper: 
 related work 2.1 text-motion retrieval: 
 Text-motion retrieval is received much attention in recent years [12, 15] , which aims to retrieve semantics relevant motion sequences by a given natural language. Different from conventional cross-modal retrieval [6, 23, 28, 29, 31] , the TMR is a challenging task due to the sequence involving multiple motions with complex semantics. In general, existing TMR methods [4, 12, 14, 15, 21, 24] follow textto-image or text-to-video retrieval, representing query and motion sequences as single-point embeddings in a common space to be retrieved based on their distance. Guo 𝑒𝑡 𝑎𝑙 . [4] adopts the triplet loss function to perform the motion retrieval, which is used for evaluate the synthesis models. Mathis 𝑒𝑡 𝑎𝑙 . [15] firstly establish the text to 3D human motion retrieval as a standalone task. They simply extend the state-of-the-art text-motion synthesis model TEMOS [14] to TMR by employing the contrastive learning widely used in information retrieval. Nicola 𝑒𝑡 𝑎𝑙 . [12] investigate the content-based large volumes of spatio-temporal skeleton data retrieval by exploiting the transformer-based approach that consists of a ViViT-based motion encoder and CLIP-based [18] text encoder. Yan 𝑒𝑡 𝑎𝑙 . [24] investigate the false negative samples that semantically similar to the anchor but are defined as the negative samples in TMR and propose a drop triplet loss function to calibrate the supervision provided by these false negative samples. However, these methods primarily focus on representing the motion sequence with complex semantics to a global representation for alignment. It inadequately captures the diverse semantics within the motion sequence and hardly enables a comprehensive cross-modal alignment.Multi-instance multi-label learning (MIML) [32] is a classical learning problem that is close to the real-world scenarios. Different from the multi-instance learning [2] and multi-label learning [30] , the MIML is a more general problem. In MIML, a sample is defined as a bag of multiple instances and associated with multiple class labels. Yang 𝑒𝑡 . 𝑎𝑙 [25] introduce the MIML into the privileged information and propose a MIML-FCN+ network to utilize the readily available privileged bags, making the system more general and practical in real world applications. Pan 𝑒𝑡 . 𝑎𝑙 [13] view the semi-supervised automatic waveform recognition as a MIML problem and propose a MIML-GAN in which a GAN is incorporated to MIML principle to establish the adversarial learning structure, through which the generator and the discriminator alternatively improve their feature representation and classification abilities, respectively. Lai 𝑒𝑡 . 𝑎𝑙 [9] introduce MIML into medical image classification and propose a broad multi-instance multi-label learning to jointly learn multiple sub-networks in a broad sense so that the diverse correlations between bags, instances, and labels can be simultaneously captured.In this work, we regard the text-motion retrieval as MIML problem and propose a multi-granularity semantics interaction that explicitly disentangle the text and motion into token, instance, and bag and exploits the graph neural networks to model correlation of these components. Through applying semantics interaction in corresponding granularity, the MGSI achieves state-of-the-art retrieval performance in two benchmark datasets. 
preliminary: 
 Given a training dataset D = {(𝑇 𝑖 , 𝑀 𝑖 )} 𝑁 𝑖=1 with 𝑁 text-motion pairs, text-motion retrieval (TMR) aims to retrieve the semantics relevant motion sequences with the query text (for clarity, we omit the sample index in the following sections). Conventional TMR represents the text and motion as the single point features 𝒕 and 𝒎While walking the person is carrying something, puts it down, and continues to walk. v n K N N h L Z G q C L + t E 4 Y 8 = " > A A A C H n i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s y I V J d F N y 4 r 2 A e 2 Y 8 m k m T Y 0 y Q x J p l C G / o V b x a 9 x J 2 7 1 a z R t B 7 H V A 4 H D O f c k N y e I O d P G d T + d 3 M r q 2 v p G f r O w t b 2 z u 1 f c P 2 j o K F G E 1 k n E I 9 U K s K a c S V o 3 z H D a i h X F I u C 0 G Q y v p 3 5 z R J V m k b w z 4 5 j 6 A v c l C x n B x k r 3 n U C k Y t L F D 1 6 3 W H L L 7 g z o L / E y U o I M t W 7 x q 9 O L S C K o N I R j r d u e G x s / x c o w w u m k 0 E k 0 j T E Z 4 j 5 t W y q x o N p P Z x t P 0 I l V e i i M l D 3 S o J n 6 O 5 F i o f V Y B H Z S Y D P Q y 9 5 U / N c L h J X R z + 2 K 2 j b s 4 w a N K D G R 0 o t r m f D S T 5 m M E 0 M l m W 8 V J h y Z C E 2 7 Q j 2 m b I q P L c F E M f s x R A Z Y Y W J s o w X b m L f c z 1 / S O C t 7 l X L l 9 r x U v c q 6 y 8 M R H M M p e H A B V b i B G t S B g I RU a u k X i t u N A 9 k 6 M S r F C c = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R l i 0 C X R j U t M 5 B F h J J 1 S o K H t T N o 7 J G T C X 7 j V + D X u j F v 9 G i 0 w M Q K e p M n J O f e 0 t y e I B D f g u l 9 O Z m 1 9 Y 3 M r u 5 3 b 2 d 3 b P 8 g f H t V N G G v K a j Q U o W 4 G x D D B F a s B B 8 G a k W Z E B o I 1 g u H N 1 G + M m D Y 8 V P c w j p g v S V / x H q c E r P T Q D m Q i J x 3 y W O r k C 2 7 R n Q G v E i 8 l B Z S i 2 s l / t 7 s h j S V T Q A U x p u W 5 E f g J 0 c C p Y J N c O z Y s I n R I + q x l q S K S G T + Z b T z B Z 1 b p 4 l 6 o 7 V G A Z + r f R E K k M W M Z 2 E l J Y G C W v a n 4 r x d I K + P f 2 z W z b d j H A Y 8 Y h V C b x b W g d + U n X E U x M E X n W / V i g S H E 0 6 5 w l 2 u b E m N L C N X c f g z T A d G E g m 0 0 Z x v z l v t Z J f V S 0 S s X y 3 c X h c p 1 2 l 0 W n a B T d I 4 8 d I k q 6 B Z V U Q 1 R p N A T e k Y v z q v z 5 r w 7 H / P R j J N m j t E C n M 8 f 9 F a i u A = = < / l a t e x i t > m 2 a < l a t e x i t s h a 1 _ b a s e 6 4 = " t F 1 v 8 I T I W p F r n V F 3 k G q h d 8 x W 7 0 4 = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R k 1 6 J L o x i U m 8 o g w k k 6 5 Q E P b m b Q d E j L h L 9 x q / B p 3 x q 1 + j R a Y G A F P 0 u T k n H v a 2 x N E n G n j u l 9 O Z m V 1 b X 0 j u 5 n b 2 t 7 Z 3 c v v H 9 R 0 G C s K V R r y U D U C o o E z C V X D D I d G p I C I g E M 9 G N x M / P o Q l G a h v D e j C H x B e p J 1 G S X G S g + t Q C R i 3 C a P 5 + 1 8 w S 2 6 U + B l 4 q W k g F J U 2 v n v V i e k s Q B p K C d a N z 0 3 M n 5 C l G G U w z j X i j V E h A 5 I D 5 q W S i J A + 8 l 0 4 z E + s U o H d 0 N l j z R 4 q v 5 N J E R o P R K B n R T E 9 P W i N x H / 9 Q J h Z f x 7 u w L b h n 3 c 4 C F Q E y o 9 v 5 b p X v k J k 1 F s Q N L Z V t 2 Y Y x P i S V e 4 w 5 R N 8 Z E l h C p m P 4 Z p n y h C j W 0 0 Z x v z F v t Z J r W z o l c q l u 4 u C u X r t L s s O k L H 6 B R 5 6 B K V 0 S 2 q o C q i S Kv F / x Q K D r 6 G S u Q = " > A A A C H H i c b V D L S g M x F M 3 4 r P V V d e k m W A R X Z U a k u i y 6 c V n B P q A d S i a 9 b U P z G J J M o Q z 9 C b e K X + N O 3 A p + j a b t I L b 1 Q O B w z j 3 J z Y l i z o z 1 / S 9 v b X 1 j c 2 s 7 t 5 P f 3 d s / O C w c H d e N S j S F G l V c 6 W Z E D H A m o W a Z 5 d C M N R A R c W h E w 7 u p 3 x i B N k z J R z u O I R S k L 1 m P U W K d 1 G x H I h W T j u o U i n 7 J n w G v k i A j R Z S h 2 i l 8 t 7 u K J g K k p Z w Y 0 w r 8 2 I Y p 0 Z Z R D p N 8 O z E Q E z o k f W g 5 K o k A E 6 a z f S f 4 3 C l d 3 F P a H W n x T P 2 b S I k w Z i w i N y m I H Z h l b y r + 6 0 X C y f j 3 d g 2 u C / e 4 x S O g V m m z u J b t 3 Y Q p k 3 F i Q d L 5 V r 2 E Y 6 v w t C n c Z d q l + N g R Q j V z H 8 N 0 Q D S h 1 v W Z d 4 0 F y / 2 s k v p l K S i X y g 9 X x c p t 1 l 0 O n a I z d I E C d I 0 q 6 B 5 V U Q 1 R x N E T e k Y v 3 q v 3 5 r 1 7 H / P R N S / L n K A F e J 8 /x y e i I g = = < / l a t e x i t > mo < l a t e x i t s h a 1 _ b a s e 6 4 = " C 1 B 2 z Y 3 j X n P l d O S y t S r Y a n t 6 j s s = " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j M i 1 W X R j c s K 9 o H t W D L p n T Y 0 y Q x J p l C G / o V b x a 9 x J 2 7 1 a z R t B 7 H V A 4 H D O f c k N y e I O d P G d T + d 3 M r q 2 v p G f r O w t b 2 z u 1 f c P 2 j o K F E U 6 j T i k W o F R A N n E u q G G Q 6 t W A E R A Y d m M L y e + s 0 R K M 0 i e W f G M f i C 9 C U L G S X G S v e d Q K R i 0 o U H r 1 s s u W V 3 B vP K r r M F Y U a j T k o W o G R A N n E m q G G Q 7 N S A E R A Y d G M L y Z + o 0 R K M 1 C e W / G E f i C 9 C X r M U q M l R 7 a g U j E p A O P p U 6 + 4 B b d G f A q 8 V J S Q C m q n f x 3 u x v S W I A 0 l B O t W 5 4 b G T 8 h y j D K Y Z J r x x o i Q o e k D y 1 L J R G g / W S 2 8 Q S f W a W L e 6 G y R x o 8 U / 8 m E i K 0 H o v A T g p i B n r Z m 4 r / e o G w M v 6 9 X Y F t w z 5 u 8 A i o C Z V e X M v 0 r v y E y S g 2 I O l 8 q 1 7 M s Q n x t C v c Z c q m + N g S Q h W z H 8 N 0 Q B S hP K r r M F Y U a j T k o W o G R A N n E m q G G Q 7 N S A E R A Y d G M L y Z + o 0 R K M 1 C e W / G E f i C 9 C X r M U q M l R 7 a g U j E p A O P 5 U 6 + 4 B b d G f A q 8 V J S Q C m q n f x 3 u x v S W I A 0 l B O t W 5 4 b G T 8 h y j D K Y Z J r x x o i Q o e k D y 1 L J R G g / W S 2 8 Q S f W a W L e 6 G y R x o 8 U / 8 m E i K 0 H o v A T g p i B n r Z m 4 r / e o G w M v 6 9 X Y F t w z 5 u 8 A i o C Z V e X M v 0 r v y E y S g 2 I O l 8 q 1 7 M s Q n x t C v c Z c q m + N g S Q h W z H 8 N 0 Q B S hQ o = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R l i 0 C X R j U t M 5 B F h J J 1 y g Y a 2 M 2 k 7 J G T C X 7 j V + D X u j F v 9 G i 0 w M Q K e p M n J O f e 0 t y e I O N P G d b + c z N r 6 x u Z W d j u 3 s 7 u 3 f 5 A / P K r r M F Y U a j T k o W o G R A N n E m q G G Q 7 N S A E R A Y d G M L y Z + o 0 R K M 1 C e W / G E f i C 9 C X r M U q M l R 7 a g U j E p A O P p U 6 + 4 B b d G f A q 8 V J S Q C m q n f x 3 u x v S W I A 0 l B O t W 5 4 b G T 8 h y j D K Y Z J r x x o i Q o e k D y 1 L J R G g / W S 2 8 Q S f W a W L e 6 G y R x o 8 U / 8 m E i K 0 H o v A T g p i B n r Z m 4 r / e o G w M v 6 9 X Y F t w z 5 u 8 A i o C Z V e X M v 0 r v y E y S g 2 I O l 8 q 1 7 M s Q n x t C v c Z c q m + N g S Q h W z H 8 N 0 Q B S hQ i G r s 1 h x K R 4 8 Z F 0 X 3 Y c z x I N Y = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R k 1 6 J L o x i U m 8 o g w k k 6 5 Q E P b m b Q d E j L h L 9 x q / B p 3 x q 1 + j R a Y G A F P 0 u T k n H v a 2 x N E n G n j u l 9 O Z m V 1 b X 0 j u 5 n b 2 t 7 Z 3 c v v H 9 R 0 G C s K V R r y U D U C o o E z C V X D D I d G p I C I g E M 9 G N x M / P o Q l G a h v D e j C H x B e p J 1 G S X G S g + t Q C R i 3 I b H 8 3 a + 4 B b d K f A y 8 V J S Q C k q 7 f x 3 q x P S W I A 0 l B O t m 5 4 b G T 8 h y j D K Y Z x r x R o i Q g e k B 0 1 L J R G g / W S 6 8 R i f W K W D u 6 G y R x o 8 V f 8 m E i K 0 H o n A T g p i + n rQ d E j L h L 9 x q / B p 3 x q 1 + j R a Y G A F P 0 u T k n H v a 2 x N E n G n j u l 9 O Z m V 1 b X 0 j u 5 n b 2 t 7 Z 3 c v v H 9 R 0 G C s K V R r y U D U C o o E z C V X D D I d G p I C I g E M 9 G N x M / P o Q l G a h v D e j C H x B e p J 1 G S X G S g + t Q C R i 3 I b H i 3 a + 4 B b d K f A y 8 V J S Q C k q 7 f x 3 q x P S W I A 0 l B O t m 5 4 b G T 8 h y j D K Y Z x r x R o i Q g e k B 0 1 L J R G g / W S 6 8 R i f W K W D u 6 G y R x o 8 V f 8 m E i K 0 H o n A T g p i + n rL F z 4 J L E = " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j M i 1 W X R j c s K 9 o H t W D L p n T Y 0 y Q x J p l C G / o V b x a 9 x J 2 7 1 a z R t B 7 H V A 4 H D O f c k N y e I O d P G d T + d 3 M r q 2 v p G f r O w t b 2 z u 1 f c P 2 j o K F E U 6 j T i k W o F R A N n E u q G G Q 6 t W A E R A Y d m M L y e + s 0 R K M 0 i e W f G M f i C 9 C U L G S X G S v e d Q K R i 0 o W H S r d Y c s v u D P g v 8 T J S Q h l q 3 e J X p x f R R I A 0 l B O t 2 5 4 b G z 8 l y j D K Y V L o J B p i Q o e k D 2 1 L J R G g / X S 2 8 Q S f W K W H w 0 j Z I w 2 e q b 8 T K R F a j 0 V g J w U x A 7 3 s T c V / v U B Y G f / c r s C 2 Y R 8 3 e A T U R E o v r m X C S z 9 l M k 4 M S D r f K k w 4 N h G e d o V 7 T N k U H 1 t C q G L 2 Y 5 g O i C L U 2 E Y L t j F v u Z + / p H F W 9 i r l y u 1 5 q X q V d Z d H R + g Y n S I P X a A q u k E 1 V E c U SF i o f V Y B H Z S Y D P Q y 9 5 U / N c L h J X R z + 2 K 2 j b s 4 w a N K D G R 0 o t r m f D S T 5 m M E 0 M l m W 8 V J h y Z C E 2 7 Q j 2 m b I q P L c F E M f s x R A Z Y Y W J s o w X b m L f c z 1 / S O C t 7 l X L l 9 r x U v c q 6 y 8 M R H M M p e H A B V b i B G t S B g I R H e I J n 5 8 V 5 d d 6 c 9 / l o z s k y h 7 A A 5 + M b 8 q W i t w = = < / l a t e x i t > m 1 a < l a t e x i t s h a 1 _ b a s e 6 4 = " U M F S m P z y U a u k X i t u N A 9 k 6 M S r F C c = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R l i 0 C X R j U t M 5 B F h J J 1 S o K H t T N o 7 J G T C X 7 j V + D X u j F v 9 G i 0 w M Q K e p M n J O f e 0 t y e I B D f g u l 9 O Z m 1 9 Y 3 M r u 5 3 b 2 d 3 b P 8 g f H t V N G G v K a j Q U o W 4 G x D D B F a s B B 8 G a k W Z E B o I 1 g u H N 1 G + M m D Y 8 V P c w j p g v S V / x H q c E r P T Q D m Q i J x 3 y W O r k C 2 7 R n Q G v E i 8 l B Z S i 2 s l / t 7 s h j S V T Q A U x p u W 5 E f g J 0 c C p Y J N c O z Y s I n R I + q x l q S K S G T + Z b T z B Z 1 b p 4 l 6 o 7 V G A Z + r f R E K k M W M Z 2 E l J Y G C W v a n 4 r x d I K + P f 2 z W z b d j H A Y 8 Y h V C b x b W g d + U n X E U x M E X n W / V i g S H E 0 6 5 w l 2 u b E m N L C N X c f g z T A d G E g m 0 0 Z x v z l v t Z J f V S 0 S s X y 3 c X h c p 1 2 l 0 W n a B T d I 4 8 d I k q 6 B Z V U Q 1 R p N A T e k Y v z q v z 5 r w 7 H / P R j J N m j t E C n M 8 f 9 F a i u A = = < / l a t e x i t > m 2a < l a t e x i t s h a 1 _ b a s e 6 4 = " s l B 6 e h e P x 2 5 s rL v F / x Q K D r 6 G S u Q = " > A A A C H H i c b V D L S g M x F M 3 4 r P V V d e k m W A R X Z U a k u i y 6 c V n B P q A d S i a 9 b U P z G J J M o Q z 9 C b e K X + N O 3 A p + j a b t I L b 1 Q O B w z j 3 J z Y l i z o z 1 / S 9 v b X 1 j c 2 s 7 t 5 P f 3 d s / O C w c H d e N S j S F G l V c 6 W Z E D H A m o W a Z 5 d C M N R A R c W h E w 7 u p 3 x i B N k z J R z u O I R S k L 1 m P U W K d 1 G x H I h W T j u o U i n 7 J n w G v k i A j R Z S h 2 i l 8 t 7 u K J g K k p Z w Y 0 w r 8 2 I Y p 0 Z Z R D p N 8 O z E Q E z o k f W g 5 K o k A E 6 a z f S f 4 3 C l d 3 F P a H W n x T P 2 b S I k w Z i w i N y m I H Z h l b y r + 6 0 X C y f j 3 d g 2 u C / e 4 x S O g V m m z u J b t 3 Y Q p k 3 F i Q d L 5 V r 2 E Y 6 v w t C n c Z d q l + N g R Q j V z H 8 N 0 Q D S h 1 v W Z d 4 0 F y / 2 s k v p l K S i X y g 9 X x c p t 1 l 0 O n a I z d I E C d I 0 q 6 B 5 V U Q 1 R x N E T e k Y v 3 q v 3 5 r 1 7 H / P R N S / L n K A F e J 8 /x y e i I g = = < / l a t e x i t > mo < l a t e x i t s h a 1 _ b a s e 6 4 = " C 1 B 2 z Y 3 j X n P l d O S y t S r Y a n t 6 j s s = " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j M i 1 W X R j c s K 9 o H t W D L p n T Y 0 y Q x J p l C G / o V b x a 9 x J 2 7 1 a z R t B 7 H V A 4 H D O f c k N y e I O d P G d T + d 3 M r q 2 v p G f r O w t b 2 z u 1 f c P 2 j o K F E U 6 j T i k W o F R A N n E u q G G Q 6 t W A E R A Y d m M L y e + s 0 R K M 0 i e W f G M f i C 9 C U L G S X G S v e d Q K R i 0 o U H r 1 s s u W V 3 B v y X e B k p o Q y 1 b v G r 0 4 t o I k A a y o n W b c + N j Z 8 S Z R j l M C l 0 E g 0 x o U P S h 7 a l k g j Q f j r b e I J P r N L D Y a T s k Q b P 1 N + J l A i t x y K w k 4 K Y g V 7 2 p u K / X i C s j H 9 u V 2 D b s I 8 b P A J q I q U X 1 z L h p Z 8 y G S c G J J 1 v F S Y c m w h P u 8 I 9 p m y K j y 0 h V D H 7 M U w H R B F q b K M F 2 5 i 3 3 M 9 f 0 j g r e 5 V y 5 f a 8 V L 3 K u s u j I 3 S M T p G H L l A V 3 a A a q i O K J H p E T + j Z e X F e n T f n f T 6 a c 7 L M I V q A 8 / E N + X G i u w = = < / l a t e x i t > m 1 e < l a t e x i t s h a 1 _ b a s e 6 4 = " A h 5 q E Y J E j 5 s g F W 6 u a Z e p 8 j p C f b k = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R l D 0 C X R j U t M 5 B F h J J 1 y g Y a 2 M 2 k 7 J G T C X 7 j V + D X u j F v 9 G i 0 w M Q K e p M n J O f e 0 t y e I O N P G d b + c z N r 6 x u Z W d j u 3 s 7 u 3 f 5 A / P K r r M F Y U a j T k o W o G R A N n E m q G G Q 7 N S A E R A Y d G M L y Z + o 0 R K M 1 C e W / G E f i C 9 C X r M U q M l R 7 a g U j E p A O P p U 6 + 4 B b d G f A q 8 V J S Q C m q n f x 3 u x v S W I A 0 l B O t W 5 4 b G T 8 h y j D K Y Z J r x x o i Q o e k D y 1 L J R G g / W S 2 8 Q S f W a W L e 6 G y R x o 8 U / 8 m E i K 0 H o v A T g p i B n r Z m 4 r / e o G w M v 6 9 X Y F t w z 5 u 8 A i o C Z V e X M v 0 r v y E y S g 2 I O l 8 q 1 7 M s Q n x t C v c Z c q m + N g S Q h W z H 8 N 0 Q B S hE = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R l j w C X R j U t M 5 B F h J J 1 y g Y a 2 M 2 k 7 J G T C X 7 j V + D X u j F v 9 G i 0 w M Q K e p M n J O f e 0 t y e I O N P G d b + c z N r 6 x u Z W d j u 3 s 7 u 3 f 5 A / P K r r M F Y U a j T k o W o G R A N n E m q G G Q 7 N S A E R A Y d G M L y Z + o 0 R K M 1 C e W / G E f i C 9 C X r M U q M l R 7 a g U j E p A O P 5 U 6 + 4 B b d G f A q 8 V J S Q C m q n f x 3 u x v S W I A 0 l B O t W 5 4 b G T 8 h y j D K Y Z J r x x o i Q o e k D y 1 L J R G g / W S 2 8 Q S f W a W L e 6 G y R x o 8 U / 8 m E i K 0 H o v A T g p i B n r Z m 4 r / e o G w M v 6 9 X Y F t w z 5 u 8 A i o C Z V e X M v 0 r v y E y S g 2 I O l 8 q 1 7 M s Q n x t C v c Z c q m + N g S Q h W z H 8 N 0 Q B S h x j a a s 4 1 5 y / 2 s k v p F 0 S s V S 3 e X h c p 1 2 l 0 W n a B T d I 4 8 V E Y V d I u q q I Y o k u g J P a M X 5 9 V 5 c 9 6 d j / l o x k k z x 2 g B z u c P A 6 a i w Q = = < / l a t e x i t > m 7 e < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 W K J z J 5 M G o H o m + 7 w w U 4 n 7 d f f w Q o = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R l i 0 C X R j U t M 5 B F h J J 1 y g Y a 2 M 2 k 7 J G T C X 7 j V + D X u j F v 9 G i 0 w M Q K e p M n J O f e 0 t y e I O N P G d b + c z N r 6 x u Z W d j u 3 s 7 u 3 f 5 A / P K r r M F Y U a j T k o W o G R A N n E m q G G Q 7 N S A E R A Y d G M L y Z + o 0 R K M 1 C e W / G E f i C 9 C X r M U q M l R 7 a g U j E p A O P p U 6 + 4 B b d G f A q 8 V J S Q C m q n f x 3 u x v S W I A 0 l B O t W 5 4 b G T 8 h y j D K Y Z J r x x o i Q o e k D y 1 L J R G g / W S 2 8 Q S f W a W L e 6 G y R x o 8 U / 8 m E i K 0 H o v A T g p i B n r Z m 4 r / e o G w M v 6 9 X Y F t w z 5 u 8 A i o C Z V e X M v 0 r v y E y S g 2 I O l 8 q 1 7 M s Q n x t C v c Z c q m + N g S Q h W z H 8 N 0 Q B S h x j a a s 4 1 5 y / 2 s k n q p 6 J W L 5 b u L Q u U 6 7 S 6 L T t A p O k c e u k Q V d I u q q I Y o k u g J P a M X 5 9 V 5 c 9 6 d j / l o x k k z x 2 g B z u c P + y K i v A = = < / l a t e x i t > m 2 e < l a t e x i t s h a 1 _ b a s e 6 4 = " H Q / Q i G r s 1 h x K R 4 8 Z F 0 X 3 Y c z x I N Y = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R k 1 6 J L o x i U m 8 o g w k k 6 5 Q E P b m b Q d E j L h L 9 x q / B p 3 x q 1 + j R a Y G A F P 0 u T k n H v a 2 x N E n G n j u l 9 O Z m V 1 b X 0 j u 5 n b 2 t 7 Z 3 c v v H 9 R 0 G C s K V R r y U D U C o o E z C V X D D I d G p I C I g E M 9 G N x M / P o Q l G a h v D e j C H x B e p J 1 G S X G S g + t Q C R i 3 I b H 8 3 a + 4 B b d K f A y 8 V J S Q C k q 7 f x 3 q x P S W I A 0 l B O t m 5 4 b G T 8 h y j D K Y Z x r x R o i Q g e k B 0 1 L J R G g / W S 6 8 R i f W K W D u 6 G y R x o 8 V f 8 m E i K 0 H o n A T g p i + n r R m 4 j / e o G w M v 6 9 X Y F t w z 5 u 8 B C o C Z W e X 8 t 0 r / y E y S g 2 I O l s q 2 7 M s Q n x p C v c Y c q m + M g S Q h W z H 8 O 0 T x S h x j a a s 4 1 5 i / 0 s k 9 p Z 0 S s V S 3 c X h f J 1 2 l 0 W H a F j d I o 8 d I n K 6 B Z V U B V R J N E T e k Y v z q v z 5 r w 7 H 7 P R j J N m D t E c n M 8 f / N O i v Q = = < / l a t e x i t > m 3 e < l a t e x i t s h a 1 _ b a s e 6 4 = " N E g D f s s w 6 1 i l t p I / u G g m V 4 b D 4 l 0 = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R m j 6 J L o x i U m 8 o g w k k 6 5 Q E P b m b Q d E j L h L 9 x q / B p 3 x q 1 + j R a Y G A F P 0 u T k n H v a 2 x N E n G n j u l 9 O Z m V 1 b X 0 j u 5 n b 2 t 7 Z 3 c v v H 9 R 0 G C s K V R r y U D U C o o E z C V X D D I d G p I C I g E M 9 G N x M / P o Q l G a h v D e j C H x B e p J 1 G S X G S g + t Q C R i 3 I b H i 3 a + 4 B b d K f A y 8 V J S Q C k q 7 f x 3 q x P S W I A 0 l B O t m 5 4 b G T 8 h y j D K Y Z x r x R o i Q g e k B 0 1 L J R G g / W S 6 8 R i f W K W D u 6 G y R x o 8 V f 8 m E i K 0 H o n A T g p i + n r R m 4 j / e o G w M v 6 9 X Y F t w z 5 u 8 B C o C Z W e X 8 t 0 r / y E y S g 2 I O l s q 2 7 M s Q n x p C v c Y c q m + M g S Q h W z H 8 O 0 T x S h x j a a s 4 1 5 i / 0 s k 9 p Z 0 S s V S 3 f n h f J 1 2 l 0 W H a F j d I o 8 d I n K 6 B Z V U B V R J N E T e k Y v z q v z 5 r w 7 H 7 P R j J N m D t E c n M 8 f A E S i v w = = < / l a t e x i t > m 5 e < l a t e x i t s h a 1 _ b a s e 6 4 = " t f u Q O S M O T 1 7 6 0 B I e + 1 + d L F z 4 J L E = " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j M i 1 W X R j c s K 9 o H t W D L p n T Y 0 y Q x J p l C G / o V b x a 9 x J 2 7 1 a z R t B 7 H V A 4 H D O f c k N y e I O d P G d T + d 3 M r q 2 v p G f r O w t b 2 z u 1 f c P 2 j o K F E U 6 j T i k W o F R A N n E u q G G Q 6 t W A E R A Y d m M L y e + s 0 R K M 0 i e W f G M f i C 9 C U L G S X G S v e d Q K R i 0 o W H S r d Y c s v u D P g v 8 T J S Q h l q 3 e J X p x f R R I A 0 l B O t 2 5 4 b G z 8 l y j D K Y V L o J B p i Q o e k D 2 1 L J R G g / X S 2 8 Q S f W K W H w 0 j Z I w 2 e q b 8 T K R F a j 0 V g J w U x A 7 3 s T c V / v U B Y G f / c r s C 2 Y R 8 3 e A T U R E o v r m X C S z 9 l M k 4 M S D r f K k w 4 N h G e d o V 7 T N k U H 1 t C q G L 2 Y 5 g O i C L U 2 E Y L t j F v u Z + / p H F W 9 i r l y u 1 5 q X q V d Z d H R + g Y n S I P X a A q u k E 1 V E c U S f S I n t C z 8 + K 8 O m / O + 3 w 0 5 2 S Z Q 7 Q A 5 + M b A f W i w A = = < / l a t e x i t > m 6e < l a t e x i t s h a 1 _ b a s e 6 4 = " j Z Z 0 x F M z W r 1 q / q h 6 9 B I v g q e y q V I 9 𝑎 and 𝒎 2 𝑎 are fused to formulate the new nodes. Finally, we exploit token, instance, and bag-wise semantics interaction to comprehensively align text and motions sequence. into a common space and exploits the cross-modal distance, 𝑒.𝑔., cosine similarity cos(𝒕, 𝒎), to measure and rank the semantics similarity achieving text-motion retrieval. However, we argue that the motion sequence consists of multiple atomic motions with diverse semantics, the simplistic point representations hardly capture the complicated semantics of motion sequence.F i t + A 6 b T O F O h o 1 c l f h x 6 o 7 E = " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j M i 1 W X R j c s K 9 o H t W D J p p g 1 N M k N y p 1 C G / o V b x a 9 x J 2 7 1 a z R t B 7 H V A 4 H D O f c k N y e I B T f g u p 9 O b m V 1 b X 0 j v 1 n Y 2 t 7 Z 3 S v u H z R M l G j K 6 j Q S k W 4 F x D D B F a s D B 8 F a s W Z E B o I 1 g + H 1 1 G + O m D Y 8 U n c w j p k v S V / x k F M C V r r v B D K F S Z c 8 e N 1 i y S 2 7 M + C / x M t I C W W o d Y t f n V 5 E E 8 k U U E G M a X t u D H 5 K N H A q 2 K T Q S Q y L C R 2 S P m t b q o h k x k 9 n G 0 / w i V V 6 O I y 0 P Q r w T P 2 d S I k 0 Z i w D O y k J D M y y N x X / 9 Q J p Z f x z u 2 a 2 D f s 4 4 B G j E G m z u B a E l 3 7 K V Z w A U 3 S + V Z g I D B G e d o V 7 X N u U G F t C q O b 2 Y 5 g O i C Y U b K M F 2 5 i 3 3 M 9 f 0 j g r e 5 V y 5 f a 8 V L 3 K u s u j I 3 S M T p G H L l A V 3 a A a q i O K F H p E T + j Z e X F e n T f n f T 6 a c 7 L M I V q A 8 / E N / p + i v g = = < / l a t e x i t > t 1 a < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 T I y y q o + V J Y s I 1 7 P f e i E d R p 1 M 3 I = " > A A A C H n i c b V D L T g I x F O 3 g C / G F u n T T S E x c k R l i 0 C X R j U t M 5 B F h J J 1 S o K H t T N o 7 J G T C X 7 j V + D X u j F v 9 G i 0 w M Q K e p M n J O f e 0 t y e I B D f g u l 9 O Z m 1 9 Y 3 M r u 5 3 b 2 d 3 b P 8 g f H t V N G G v K a j Q U o W 4 G x D D B F a s B B 8 G a k W Z E B o I 1 g u H N 1 G + M m D Y 8 V P c w j p g v S V / x H q c E r P T Q D m Q C k w 5 5 L H X y B b f o z o B X i Z e S A k p R 7 e S / 2 9 2 Q x p I p o I I Y 0 / L c C P y E a O B U s E m u H R s W E T o k f d a y V B H J j J / M N p 7 g M 6 t 0 c S / U 9 i j A M / V v I i H S m L E M 7 K Q k M D D L 3 l T 8 1 w u k l f H v 7 Z r Z N u z j g E e M Q q j N 4 l r Q u / I T r q I Y m K L z r X q x w B D i a V e 4 y 7 V N i b E l h G p u P 4 b p g G h C w T a a s 4 1 5 y / 2 s k n q p 6 J W L 5 b u L Q u U 6 7 S 6 L T t A p O k c e u k Q V d I u q q I Y o U u g J P a M X 5 9 V 5 c 9 6 d j / l o x k k z x 2 g B z u c P A F + i v w = = < / l a t e x i t > t 2 a < l a t e x i t s h a 1 _ b a s e 6 4 = " a d 1 h c W T E M V p t X Z h T e S 2 O Y v A E D 6 4 = " > A A A C H H i c b V D L S g M x F M 3 4 r P V V d e k m W A R X Z U a k u i y 6 c V n B P q A d S i Z N 2 9 A 8 h u R O o Q z 9 C b e K X + N O 3 A p + j a b t I L b 1 Q O B w z j 3 J z Y l i w S 3 4 / p e 3 t r 6 x u b W d 2 8 n v 7 u 0 f H B a O j u t W J 4 a y G t V C m 2 Z E L B N c s R p w E K w Z G 0 Z k J F g j G t 5 N / c a I G c u 1 e o R x z E J J + o r 3 O C X g p G Y 7 k i l M O r p T K P o l f w a 8 S o K M F F G G a q f w 3 e 5 q m k i m g A p i b S v w Y w h T Y o B T w S b 5 d m J Z T O i Q 9 F n L U U U k s 2 E 6 2 3 e C z 5 3 S x T 1 t 3 F G A Z + r f R E q k t W M Z u U l J Y G C X v a n 4 r x d J J + P f 2 w 1 z X b j H A Y 8 Y B W 3 s 4 l r Q u w l T r u I E m K L z r X q J w K D x t C n c 5 c a l x N g R Q g 1 3 H 8 N 0 Q A y h 4 P r M u 8 a C 5 X 5 W S f 2 y F J R L 5 Y e r Y u U 2 6 y 6 H T t E Z u k A B u k Y V d I + q q I Y o E u g J P a M X 7 9 V 7 8 9 6 9 j / n o m p d l T t A C v M 8 f 0 x O i K Q = = < / l a t e x i t > t o < l a t e x i t s h a 1 _ b a s e 6 4 = " f A C b w 5 u p 9 k h + l 7 s X D w X m 3 q W E z 8 8 = " > A A A C H n i c b V B N S w M x F M z 6 W e t X 1 a O X Y B E 8 l V 0 R 9 V j 0 4 r G C 1 a J d S z Z 9 2 w a T 7 J K 8 F c r S f + F V 8 d d 4 E 6 / 6 a z R t F 7 H q Q G C Y e Z O 8 T J R K Y d H 3 P 7 y Z 2 b n 5 h c X S U n l 5 Z X V t v b K x e W m T z H B o 8 k Q m p h U x C 1 J o a K J A C a 3 U A F O R h K v o 7 n T k X 9 2 D s S L R F z h I I V S s p 0 U s O E M n X b c j l e O w A 7 d B p 1 L 1 a / 4 Y 9 C 8 J C l I l B R q d y m e 7 m / B M g U Y u m b U 3 g Z 9 i m D O D g k s Y l t u Z h Z T x O 9 a D G 0 c 1 U 2 D D f L z x k O 4 6 p U v j x L i j k Y 7 V n 4 m c K W s H K n K T i m H f / v Z G 4V G o 6 + X u X J v v 0 = " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j N F 1 G X R j c s K 9 o F t L Z n 0 1 o Y m m S G 5 I 5 S h f + F W 8 W v c i V v 9 G k 0 f i K 0 e C B z O u S e 5 O W E s h U X f / / Q y S 8 s r q 2 v Z 9 d z G 5 t b 2 T n 5 3 r 2 a j x H C o 8 k h G p h E y C 1 J o q K J A C Y 3 Y A F O h h H o 4 u B z 7 9 Q c w V k T 6 B o c x t B W 7 1 6 I n O E M n 3 b Z C l e K o A 3 e l T r 7 g F / 0 J 6 F 8 S z E i B z F D p 5 L 9 a 3 Y g n C j R y y a x t B n 6 M 7 Z Q Z F F z C K N d K L M S M D 9 g 9 N B 3 V T I F t p 5 O N R / T I K V 3 a i 4 w 7 G u l E / Z 1 I m b J 2 q E I 3 q R j 2 7 a I 3 F v / 1 Q u V k + n O 7 A d e G e x z p A 3 C M j J 1 f C 3 v n 7 V T o O E H Q f L p V L 5 E U I z r u i n a F c S k 5 d I R x I 9 z H K O 8 z w z i 6 R n O u s W C x n 7 + k V i o G p 8 X T 6 5 N C + W L W X Z Y c k E N y T A J y R s r k i l R I l X C i y S N 5 I s / e i / f q v X n v 0 9 G M N 8 v s k z l 4 H 9 8 H K 6 L D < / l a t e x i t > t 2 e < l a t e x i t s h a 1 _ b a s e 6 4 = " V b K S 2 H 3 i m 4 4 x U c 3 I s / O f P a W v G T w = " > A A A C H n i c b V B N S w M x F M z W r 1 q / q h 6 9 B I v g q e y q V I 9 F L x 4 r W B X b t W T T V w 0 m 2 S V 5 W y h L / 4 V X x V / j T b z q r 9 G 0 L m K t A 4 F h 5 k 3 y M l E i h U X f / / A K M 7 N z 8 w v F x d L S 8 s r q W n l 9 4 8 L G q e H Q 5 L G M z V X E L E i h o Y k C J V w l B p i K J F x G 9 y c j / 7 I P x o p Y n + M g g V C x W y 1 6 g j N 0 0 n U 7 U h k O O 3 C z 3 y l X / K o / B p 0 m Q U 4 q J E e j U / 5 s d 2 O e K t D I J b O 2 F f g J h h k z K L i E Y a m d W k g Y v 2 e 3 0 H J U M w U 2 z M Y b D + m O U 7 q 0 F x t 3 N N K x + j u R M W X t Q E V u U j G 8 s 3 + 9 k f i v F y k n 0 5 / b D b g 2 3 O N I + 8 A x N n Z y L e w d h Z n Q S Y q g + f d W v V R S j O m o K 9 o V x q X k w B H G j X A f o / y O G c b R N V p y j Q V / + 5 k m F 3 v V o F a t n= " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j N S 1 G X R j c s K 9 o F t L Z n 0 1 o Y m m S G 5 I 5 S h f + F W 8 W v c i V v 9 G k 0 f i K 0 e C B z O u S e 5 O W E s h U X f / / Q y S 8 s r q 2 v Z 9 d z G 5 t b 2 T n 5 3 r 2 a j x H C o 8 k h G p h E y C 1 J o q K J A C Y 3 Y A F O hG V E G x z J q 0 i G Y F d R D X C 5 + 1 n x Y = " > A A A C A X i c b V B N S 8 N A E N 3 U r 1 q / o l 4 E L 8 E i e C q J F P V Y 9 O L B Q w X 7 A U 0 o m + 2 m X b r Z h N 2 J W E K 8 + Fi k o R U A O F Y q Z 5 j x + C l W A I j n G Y V N 1 E 0 x m S M h 7 S n q c A h V V 4 6 / S C z j r U y s I J I 6 h J g T d X f E y k O l Z q E v u 7 M 7 1 X z X i 7 + 5 / U S C C 6 8 l I k 4 A S r I b F G Q c A s i K 4 / D G j B J C f C J J p h I p m + 1 y A h L T E C H V t E h O P M v L 5 L 2 a c 0 5 q 9 V v 6 9 X G Z R F H G R 2 i I 3 S C H H S O G u g a N V E L E f S I n t EQ p k 3 F i Q d L 5 V r 2 E Y 6 v w t C n c Z d q l + N g R Q j V z H 8 N 0 Q D S h 1 v W Z d 4 0 F y / 2 s k v p l K S i X y g 9 X x c p t 1 l 0 O n a I z d I E C d I 0 q 6 B 5 V U Q 1 R x N E T e k Y vF i t + A 6 b T O F O h o 1 c l f h x 6 o 7 E = " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j M i 1 W X R j c s K 9 o H t W D J p p g 1 N M k N y p 1 C G / o V b x a 9 x J 2 7 1 a z R t B 7 H V A 4 H D O f c k N y e I B T f g u p 9 O b m V 1 b X 0 j v 1 n Y 2 t 7 Z 3 S v u H z R M l G j K 6 j Q S k W 4 F x D D B F a s D B 8 F a s W Z E B o I 1 g + H 1 1 G + O m D Y 8 U n c w j p k v S V / x k F M C V r r v B D K F S Z c 8 e N 1 i y S 2 7 M + C / x M t I C W W o d Y t f n V 5 E E 8 k U U E G M a X t u D H 5 K N H A q 2 K T Q S Q y L C R 2 S P m t b q o h k x k 9 n G 0 / w i V V 6 O I y 0 P Q r w T P 2 d S I k 0 Z i w D O y k J D M y y N x X / 9 Q J p Z f x z u 2 a 2 D f s 4 4 B G j E G m z u B a E l 3 7 K V Z w A U 3 S + V Z g I D B G e d o V 7 X N u U G F t C q O b 2 Y 5 g O i C Y U b K M Fg f H t V N G G v K a j Q U o W 4 G x D D B F a s B B 8 G a k W Z E B o I 1 g u H N 1 G + M m D Y 8 V P c w j p g v S V / x H q c E r P T Q D m Q i J x 3 y W O r k C 2 7 R n Q G v E i 8 l B Z S i 2 s l / t 7 s h j S V T Q A U x p u W 5 E f g J 0 c C p Y J N c O z Y s I n R I + q x l q S K S G T + Z b T z B Z 1 b p 4 l 6 o 7 V G A Z + r f R E K k M W M Z 2 E l J Y G C W v a n 4 r x d I K + P f 2 z W z b d j H A Y 8 Y h V C b x b W g d + U n X E U x M E X n W / V i g S H E 0 6 5 w l 2 u b E m N L C N X c f g z T A d G E g m 0 0 Z x v z l v t Z J f V SM f V C P B R s w A g G L f n m g R t i G B H M 0 5 v M T 1 2 g D 5 A y o b L M N 6 t 2 z Z 7 C W i R O Q a q o Q N M 3 v 9 1 + R J K Q C i A c K 9 V z 7 B i 8 F E t g h N OF i t + A 6 b T O F O h o 1 c l f h x 6 o 7 E = " > A A A C H n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j M i 1 W X R j c s K 9 o H t W D J p p g 1 N M k N y p 1 C G / o V b x a 9 x J 2 7 1 a z R t B 7 H V A 4 H D O f c k N y e I B T f g u p 9 O b m V 1 b X 0 j v 1 n Y 2 t 7 Z 3 S v u H z R M l G j K 6 j Q S k W 4 F x D D B F a s D B 8 F a s W Z E B o I 1 g + H 1 1 G + O m D Y 8 U n c w j p k v S V / x k F M C V r r v B D K F S Z c 8 e N 1 i y S 2 7 M + C / x M t I C W W o d Y t f n V 5 E E 8 k U U E G M a X t u D H 5 K N H A q 2 K T Q S Q y L C RF L x 4 r W B X b t W T T V w 0 m 2 S V 5 W y h L / 4 V X x V / j T b z q r 9 G 0 L m K t A 4 F h 5 k 3 y M l E i h U X f / / A K M 7 N z 8 w v F x d L S 8 s r q W n l 9 4 8 L G q e H Q 5 L G M z V X E L E i h o Y k C J V w l B p i K J F x G 9 y c j / 7 I P x o p Y n + M g g V C x W y 1 6 g j N 0 0 n U 7 U h k O O 3 C z 3 y l X / K o / B p 0 m Q U 4 q J E e j U / 5 s d 2 O e K t D I J b O 2 F f g J h h k z K L i E Y a m d W k g Y v 2 e 3 0 H J U M w U 2 z M Y b D + m O U 7 q 0 F x t 3 N N K x + j u R M W X t Q E V u U j G 8 s 3 + 9 k f i v F y k n 0 5 / b D b g 2 3 O N I + 8 A x N n Z y L e w d h Z n Q S Y q g + f d W v V R S j O m o K 9 o V x q X k w B H G j X A f o / y O G c b R N V p y j Q V / + 5 k m F 3 v V o F a t nV t f x 6 Y W N z a 3 u n u L v X 0 G G s G N R Z K E L V 8 q g G w S X U k a O A V q S A B p 6 A p j e 6 y v z m P S j N Q 3 m H 4 w i 6 A R 1 I 7 n N G 0 U i 9 4 k E n o D h k V C Q 3 a S / p I D x g g u E I Z J r 2 i i W n 7 E x g L x J 3 R k p k h l q v + N X p h y w O Q C I T V O u 2 6 0 T Y T a h C z g S k h U 6 s I a J s R A f Q N l T S A H Q 3 m f y Q 2 s d G 6 d t + q E x J t C f= " > A A A B / X i c b V A 9 S w N B F H w X v 2 L 8 i l r a L A b B K t x J U M u g j Y 0 Q w V w C y R H 2 N n v J m t 2 9 Y 3 d P C E f w L 9 h q b y e 2 / h Z b f 4 m b 5 A p N H H g w z L z H P C Z M O N P G d b + c w s r q 2 v p G c b O 0 t b 2 z u 1 f e P / B 1 n C p C m y T m s W q H W F P O J G 0 a Z j h t J 4 p i E X L a C k f X U 7 / 1 S J V m s b w 3 4 4 Q G A g 8 k i x j B x k p + N x T Z 7 a R X r r h V d w a 0 T L y c V C B H o 1 f + 7 v Z j k g o q D e F Y 6 4 7 n J i b I s D K M c D o p d V N N E 0 x G e E A 7 l k o s q A 6 y 2 b c T d G K V P o p i Z U c a N F N / X 2 R Y a D 0 W o d 0 U 2 A z 1 o j c V / / V C s Z B s o s s g Y z J J D Z V k H h y l H J k Y T a t A f a Y o M X x s C S a K 2 d 8 R G W K F i b G F l WO R o E u D y o o F U N N X W L y E F U X Q = " > A A A C B H i c b V C S g N B F J N r x h f U U u b w S B Y h V R a B m D K C e U C y h N n J b D J k Z n a d u S u E J a / Y K u n d j H Z + i b P J F p p M L h n H u h x P E g h t w S + n s L K t r R C x t b e / s p X D o m S j R l T R q J S H c C Y p j g i j W B g C d W D M i A H a w f g u P T B s e q X u Y x M y X Z K h y C k B K / k S W B E i U h v p n o l y t u Z B L x M v J x W U o E v f / c G E U k U A F M a b r u T H K d H A q W D T U i x L C Z T I a s a k i k h k / n Y W e h O r D H A Y a T s K E z f Z E S a c x E B n Y z C k W v U z w v k w m c I L / U q z g B p u j c Z g I D B H O G s E D r h k F M b G E U M t d k x H R B M K t r e S L c V b r G C Z t M q X q a u z u v K / y e o r o C B j U + S h C R H t i B m o i i B / S M X t C r + S O e / O x y O Q h + g P n M f l g u Y Q = = < / l a t e x i t > G t < l a t e x i t s h a 1 _ b a s e 6 4 = " z s p f y Q I o C U b m Y F g f 4 p u 0 d j Y 7 6 W Q = " > A A A C B H i c b V C 7 S g N B F L 0 b X z G + o p Y 2 g 0 G w C r s i 0 T J o o W U E 8 4 B k C b O T 2 W T I z O w 6 M y u E J a 2 / Y K u 9 n d j 6 H 7 Z + i b P J F p p 4 4 M L h n H u 5 h x P E n G n j u l 9 O Y W V 1 b X 2 j u F n a 2 t 7 Z 3 S v v H 7 R 0 l C h C m y T i k e o E W F P O J G 0 a Z j j t x I p i E X D a D s b X m d 9 + p E q z S N 6 b S U x 9 g Y e S h Y x g Y y W / J 7 A Z E c z T m 2 l f 9 M s V t + r O g J a J l 5 M K 5 G j 0 y 9 + 9 Q U Q S Q a U h H G v d 9 d z Y + C l W h h F O p 6 V e o m m M y R g P a d d S i Q X V f j o L P U U n V h m g M F J 2 p E E z 9 f d F i o X W E x H Y z S y k X v Q y 8 V 8 v E A u f T X j p p 0 z G i a G S z B + H C U c m Q l k j a M A U J Y Z P L M F E M Z s d k R F W m B j b W 8 m W 4 i 1 W s E x a Z 1 W v V q 3 d n V f q V 3 k 9 R T i C Y z g F D y 6 g D r f Q g C Y Q e I B nIn this work, we formulate the text-motion retrieval as the multiinstance multi-label learning (MIML) problem, where the motion sequence and query are viewed as the bags of multiple instances. For motion and text bags, the instances are viewed as the atomic motions and corresponding descriptions. To solve the MIML problem, we propose a novel multi-granularity semantics interaction (MGSI) approach that exploits the graph neural networks (GNNs) to model the text and motion sequence into different levels, performing the corresponding level semantics interaction to achieve a precise alignment and considerable performance. Specifically, we decompose the text and motion sequences into different level and represent them as the graph G 𝑡 = {𝑽 𝑡 , 𝑬 𝑡 } and G 𝑚 = {𝑽 𝑚 , 𝑬 𝑚 }, where 𝑽 𝑡 = {𝒕 𝑜 , {𝒕 𝑖 𝑎 } N 𝑎 𝑖=1 , {𝒕 𝑖 𝑒 } N 𝑡 𝑒 𝑖=1 } and 𝑽 𝑚 = {𝒎 𝑜 , {𝒎 𝑖 𝑎 } N 𝑎 𝑖=1 , {𝒎 𝑖 𝑒 } N 𝑚 𝑒 𝑖=1 }, 
methodology 4.1 text graph construction: 
 Given a natural language description 𝑇 = {𝑡 1 , • • • , 𝑡 N 𝑡 } with N 𝑡 words, we adopt a pre-trained frozen CLIP text encoder [18] to encode 𝑇 as a word sequences𝑻 = {𝒕 1 , • • • , 𝒕 N 𝑡 }, where 𝒕 * ∈ R d 𝑡 .We perform mean pooling for 𝑻 to initialize the 𝒕 𝑜 . Then, we adopt the off-the-shelf semantic role parser [20] to extract noun phrases and verbs from 𝑻 as well as their semantic role 𝑬 𝑡 of each noun phrase. The verbs representations are leveraged as the instance nodes {𝒕 𝑖 𝑎 } N 𝑎 𝑖=1 and connected with the root node 𝒕 𝑜 with direct edges. The noun phrases are used as the token nodes {𝒕 𝑖 𝑒 }N 𝑡 𝑒 𝑖=1 and connected with corresponding instance nodes 𝒕 * 𝑎 , where the edge 𝒆 𝑖,𝑗 between 𝑖-th token node and 𝑗-th instance node is represented by the semantics role of the token about the motion. Considering that multiple atomic motions may occur simultaneously to the same token node, we duplicate the token node for each semantic role and connect them with corresponding motion nodes. In Fig. 2 , we also show the example of the constructed text graph. For the motion sequence 𝑀, we adopt the the model in previous work [4] to extract the skeleton features 𝑴 𝑠 ∈ R N 𝑚 ×J×d 𝑠 , where J and d 𝑠 are the number and feature dimension of skeleton nodes, respectively. Then, the SMPL [10] is adopted to extract the framelevel representations𝑴 𝑓 = {𝒎 1 , • • • , 𝒎 N 𝑓 } from 𝑴 𝑠, where the N 𝑓 indicates the frame number. Before constructing the instance node, we first downsample the 𝑴 𝑓 in the temporal domain to reduce the length of the feature sequence. It reduces the computational complexity while maintaining comparable performance. For motion sequence representation 𝑴 𝑓 ∈ R N 𝑓 ×𝑑 𝑚 , we downsample it into a fixed number of features 𝑴 𝑑 ∈ R N 𝑚 ×𝑑 𝑚 by conducting the mean pooling to prevent lose information of these reduced frames, where N 𝑚 < N 𝑓 is the number of frames after downsampling. Graph Initialization. We apply mean pooling to aggregate semantics from 𝑴 𝑓 and get the sequence-level representations to initialize 𝒎 𝑜 . For the instance nodes, considering the motions occur sequentially, we adopt the simple yet effective slide windows strategy [3] to construct the atomic motion instance. Specifically, we set up multiple slide windows of different lengths with a stride of 1 and perform overlapping as the sliding windows move. The windows size set to 𝑤 = {1, 2, • • • , N 𝑚 }. Given a sliding window 𝑘, a clip feature is obtained by mean pooling over the features within 𝑘. The motion sequences could be split as the motion clips𝑴 𝑐 = {𝒎 𝑖 𝑐 } N 𝑐 𝑖=1 , where 𝑴 𝑐 ∈ R N 𝑐 ×𝑑 𝑚 and N 𝑐 = N 𝑚 (N 𝑚 +1) 2. To remove the redundant clips from 𝑴 𝑐 , we exploit the previously constructed probabilistic embedding space [14] to filter the clips:EQUATIONwhere t𝑖 𝑎 and m𝑗 𝑐 are the 𝑖-th noun phrase and 𝑗-th clips that both are extracted by the pre-trained model [14] . Then, we adopt {𝒎 1 𝑎 , • • • , 𝒎 𝑘 𝑎 } to initialize the embeddings of instance nodes. For token node 𝒎 𝑖 𝑒 , we directly adopt the frame-level representations of each atomic motion as the embeddings to initialize. The instance nodes connect to the root and token nodes with edges that is calculated by:EQUATIONwhere cos(•, •) indicates the cosine similarity. The * ∈ {𝑒, 𝑜 } and 𝑟 indicates the node index associated with node 𝑖.In TMR, the atomic motion in the sequence may be co-occurring and coupled together. Directly representing these semantically complex data samples as point embeddings and performing alignment may be unable to capture the abundant semantics and disturb the crossmodal alignment affecting the retrieval performance. To address this challenge, we introduce a co-occurring motion mining approach by measuring the semantic consistency between atomic motions. Then, we fuse the identified co-occurring motions to generate a new co-occurrence motion representation. Specifically, given the motion graph G 𝑚 = {𝑽 𝑚 , 𝑬 𝑚 }, the token nodes {𝒎 𝑥 𝑒 }N 𝑡 𝑒 𝑥=1 connected to the instance nodes 𝒎 𝑖 𝑎 are used for calculating the semantics consistency score C 𝑖,𝑗 :EQUATIONwhere |𝒎 𝑖 𝑎 | and |𝒎 𝑗 𝑎 | indicate the degree of 𝒎 𝑖 𝑎 and 𝒎 𝑗 𝑎 , respectively. The Eq. 3 draws inspiration from the principle that co-occurring motions may encapsulate as many semantically similar frames as possible. After calculating the semantics consistency score, we empirically set a threshold 𝜆 to mine the co-occurring motions. If the semantics consistency score C 𝑤𝑜𝑖,𝑗 is larger than the threshold 𝜆, the atomic motions 𝑖 and 𝑗 are identified as the co-occurring motions, otherwise are considered as the motions that occurred sequentially. For these co-occurring motions, the nodes 𝒎 𝑖 𝑎 and 𝒎are fused to obtain the co-occurrence motions nodes by adding and all token nodes belonging to 𝒎 𝑖 𝑎 and 𝒎 𝑗 𝑎 are connected to the new instance nodes. As shown in Fig. 2 (b ), the C 1,2 is larger than the threshold 𝜆. Therefore, the instance nodes 𝒎 1 𝑎 and 𝒎 1 𝑎 are merged as the newly instance nodes 𝒎 1 𝑎 , where all corresponding token nodes are connected with the new nodes. Similarly, the corresponding instance nodes in the textual graph G 𝑡 are fused to guarantee the structure consistent with G 𝑚 . Notable, we utilize the residual connection in graph reasoning to reduce the redundancy information introduced by these token nodes.Text Graph Reasoning. Considering the multiple semantics role involved in text, we adopt the rational graph convolutional network (R-GCN) [19] to model correlations of these nodes. Specifically, considering the existence of two types of nodes in the text graph, we adopt 2-layer graph convolution networks with residual connection to capture the semantics of nodes. Based on the initialized nodes 𝑽 𝑡 𝑖 = {𝒕 𝑜 , {𝒕 𝑖 𝑎 } N 𝑎 𝑖=1 ,where the 𝑾 𝑚, * 𝑟 are the learnable parameters of the motion graph. The 𝑅 is number of relations corresponding to node 𝑖. Token-wise Interaction. Besides the instance and bag-wise semantics interaction, we introduce a toke-wise interaction to provide fine-grained semantics alignment as complementary. As shown in Fig. 2 (b), we add the position embeddings 𝒑 𝑡 and 𝒑 𝑚 to the wordand frame-level representaions 𝑻 and 𝑴 and feed them into a transformer encoder:EQUATIONwhere [•; •] is the concatenate operation. Φ 𝑡 is the 2-layer transformer encoder. We further adopt multi layer perception (MLP) with ReLU activation function [1] to calculate the similarity:EQUATIONwhere the MLP consists of two linear layers with ReLU activation functions. Instance-wise Interaction. We adopt the representation of instance nodes from the text and motion graph to perform the instance-wise semantics interaction. Specifically, given the {𝒕 𝑖 𝑎 } N 𝑎 𝑖=1 and {𝒎 𝑖 𝑎 } N 𝑎 𝑖=1 , the instance-wise similarity is calculated by:EQUATIONBag-wise Interaction. For the bag-wise semantics interaction, we directly conduct the semantic interaction between the root node representations of text and motion graph:EQUATIONTraining. In our proposed multi-granularity multi-instance learning, the positive pair is defined as the motion containing certain content relevant to the query text. The negative pairs are those without any relevant content. We adopt the InfoNCE loss [28, 29] function that is widely used in the retrieve related tasks as the training objective function over the mini-batch 𝐵: (10) where S(•, •) is the similarity measurement, 𝑒.𝑔. cosine similarity. The token-wise semantics interaction is defined as follows:L info = - 1 |𝐵| ∑︁ 𝒙 𝑖 ,EQUATION)where CrossEn is the cross entropy loss function. Thus, the total training loss function is:EQUATION)where L ins and L bag indicate applying the instance-wise similarity S ins and bag-wise similarity S bag to the Eq 10, respectively.Inference. After the model is converged, the similarity between query text 𝑇 and motion sequence 𝑀 is computed by a combination of the instance-wise, bag-wise, and token-wise similarities: S(𝑇 , 𝑀) = 1 3 S token + S ins + S bag (13)  
experiments 5.1 datasets: 
 We validate the proposed methods on the two widely used 3D human motion datasets: HumanML3D [4] and KIT Motion-Language Datasets [16] : HumanML3D [4] (HumanML3D) is currently the largest 3D human motion dataset with textual descriptions. The motion sequences are originally from two already-existing and widely-used motioncapture datasets AMASS [11] and HumanAct12 [5] . Following the benchmark [15] , we split the train, validation, and test set with 23384, 1460, and 4380 motions. Each motion sequence contains approximately 3 text descriptions with different lengths. KIT Motion-Language [16] (KIT-ML) contains 3,911 recordings of full body motion and 6,278 text descriptions. Each motion sequence is described in 1 to 4 texts. The average length of text descriptions is approximately 8. Following the setup in the benchmark [15] , we adopt 4,888, 300, and 800 motion sequences as the training, validation, and test set, respectively.Baselines. We provide the comprehensive comparison with the state-of-the-art approaches, including TEMOS (ECCV2022) [14] , MotionCLIP (ECCV2022) [21] , T2M (CVPR2022) [4] , DTL (MM Asia2023) [24] , TMR (ICCV2023) [15] , and MoT (SIGIR2023) [12] .The MotionCLIP [21] , T2M [4] , and MoT [12] learn the text and motion into a common space, directly measuring the cosine similarity between global embeddings for alignment. The TEMOS [14] and TMR [15] employ a VAE structure to learn the text and motion into latent space while adopting the reparameterization technique to sample the representations from distributions for alignment [14, 15] .The DTL [24] adopts a dual-branch unimodal network to extract motion and text embeddings and project them into a common embeddings space. However, the DTL splits the HumanML3D and KIT-ML by themselves and the scale of test sets is far less than the splits used in our work. To compare the results fairly, we adopt the open-source code of DTL to train the model in our splits and report the results. The other results of baselines in our work are from their official reports. Metrics. We adopt the common metrics to report retrieval performance, including Recall at K (R@K), Median Rank (MedR), and Rsum. The R@K is the fraction of queries that correctly retrieve desired items in the top K of the ranking list. Following the benchmark [15] , K = 1,2,3,5,10 are adopted. The MedR computes the median rank of the correct targets for a query. Additionally, we report the Rsum metric which is calculated by the summing of R@K values. It evaluates retrieval performance from an overall perspective. In all tables, the metric with an upward arrow (denoted by ↑) signifies that a higher value correlates with better performance (R@K, Rsum), while the downward arrow (denoted by ↓) indicates that lower values represent superior performance (MedR). The best evaluation results are highlighted in "bold".In this work, we adopt AdamW [8] as the optimizer with a 1e-4 learning rate and set the batch size to 64 on all datasets. The text Table 1 : Performance comparison with the state-of-the-art methods on HumanML3D [4] . The "Text → Motion" indicates text-to-motion retrieval and "Motion → Text" is the motion-to-text retrieval, respectively. GPUs with 80GB memory. To enable a consistent comparison with the baseline, we follow the settings of previous work [15] to randomly select a text as the matching text for training and adopt the first text in the test set to report the evaluation performance.Text → Motion Motion → Text Rsum ↑ R@1 ↑ R@2 ↑ R@3 ↑ R@5 ↑ R@10 ↑ MedR ↓ R@1 ↑ R@2 ↑ R@3 ↑ R@5 ↑ R@10 ↑ MedR ↓ T2M [EQUATIONIn this subsection, we show the experimental results of the proposed MGSI and the state-of-the-art (SOTA) methods of TMR on the HumanML3D [4] and KIT-ML [16] . In Tab. 1, we observe that the baselines [4, 12, 14, 21] show an unsatisfying performance. The conventional approaches roughly represent the query and motion sequences as a single point to perform a global-level alignment, which are based on the assumption that the data instance in TMR only involves unique semantics. However, in reality, there exist many different motions in the query and motions sequences. Such simplistic learning strategies may difficult to capture the complex semantics resulting in an unsatisfying performance especially on the fine-grained retrieval (R@1). In our work, we formulate the TMR as the multi-instance multi-label learning trying to decompose the coupled semantics in the query and motion sequences and perform semantics alignment in corresponding levels. The significant improvement (23.09% in Rsum) of our MGSI in HumanML3D proves the effectiveness of our methods.For the KIT-ML dataset, it shows that our MGSI surpasses the current SOTA TMR approaches across all evaluation protocols with a clear margin. Especially on the Rsum metrics, our method outperforms the SOTA work TMR [15] by 21.84%. Since these baselines focus on the whole similarity between queries and motion sequences, the result indicates that such coarse-grained global similarity modeling is sub-optimal for TMR. It demonstrates the superiority of our hierarchical semantics alignment in text to 3D human motion retrieval.The effectiveness of semantics interactions. In this subsection, we verify the contributions of the different levels semantics interactions in the proposed multi-instance multi-label learning. As shown in Tab. 3, we observe that 1) when applying the bag-wise semantics interaction, the model shows the worst performance, which is close to the baselines solely aligning the point embeddings of text and motion (in Tab. 1 and Tab. 2). It proves that the simplistic global semantics alignment is unsuitable for the TMR. 2) When incorporating the instance-or token-wise with bag-wise Table 3 : Ablation study of different semantics interaction on HumanML3D and KIT-ML. We report the results on the text-to-motion retrieval. semantics interaction, the model achieves a considerable retrieval performance. The improvements achieved by the instance-and token-wise semantics interaction indicate the necessity of aligning the query text and motion sequence in a fine-grained scale.EQUATIONMethods Text → Motion Motion → Text R@1 ↑ R@2 ↑ R@3 ↑ R@1 ↑ R@2 ↑ R@3 ↑ w/3) The improvements brought by the instance-wise are larger than the token-wise. Considering the redundant information in token, directly concatenating all tokens of text and motion without filtering may introduce too much useless information to disturb the crossmodal aligning. The proposed instance-wise semantics interaction exploits the specifically designed downsample and co-occurring motion mining to refine the semantics within motion sequences bringing much retrieval performance. 4) The complete version of our method, incorporating bag-, instance-, and token-wise semantics interaction, shows the best retrieval performance. This proves that our approach that focuses on semantics interactions at different level is complementary to each other.The effectiveness of components. To examine the usefulness of the specific designed strategies in in MGIS, we compare the counterpart without the downsample or the motion fusion on the Hu-manML3D. As shown in Tab. 4, we observe that 1) when removing the downsample strategy, the retrieval results are further boosted but limited. Adopting all frame of motion sequences may introduce significant computing costs. Therefore, we conduct and report all experiment results on the downsampling version.2) The retrieval performance degenerates when we detach the co-occurrence motion fusion and straightly align the initialized instance nodes in the text and motion graph, which proves the effectiveness of our motion fusion strategy.The effectiveness of clip selection. In this subsection, we aim to verify the clip selection strategy in Sec. 4.2. The results are shown in Tab. 5. The "Random" indicates that randomly selecting the clips from 𝑴 𝑐 constructs the instance nodes in the motion graph. The results show that the clips 𝑴 𝑐 selected by the sliding window contain useless semantics for retrieval. The simplistic random selection criterion in mining the atomic motion is ineffective. The proposed clips filter strategy in Eq. 1 utilizes the pre-constructed cross-modal probabilistic space to find semantically similar clips to the action phrases. The performance improvement compared to "Random" proves that the proposed methods could effectively filter the irrelevant clips.Text → Motion Motion → Text R@1 ↑ R@2 ↑ R@3 ↑ R@1 ↑ R@2 ↑ R@Hyperparameter analysis. In this subsection, we investigate the influence of hyperparameters 𝜆 on the HumanML3D. The 𝜆 is the threshold to identify the co-occurrence motions. If the semantic consistency score is greater than 𝜆, these actions are considered co-occurring motions. As shown in Fig. 4 , we set the 𝜆 from 0.1 to 0.9. The results lead to several observations: 1) when 𝜆 is small, the retrieval performance remains lower and improves with the increase of 𝜆, which proves that a loose criterion is insufficient to identify the accurate co-occurrence motions. The co-occurrence motions should maintain as many semantically consistent frames as possible. 2) When 𝜆 exceeds the threshold of 0.8, there is a noticeable decline in retrieval performance. The higher value set for 𝜆 may overly stress the semantic consistency, imposing an overly strict criterion. It could limit the detection of co-occurring motions, potentially reducing the retrieval performance.In Fig. 3 , we visualize the retrieval results for text-to-motion retrieval of the proposed MGSI and the state-of-the-art TMR [15] on the HumanML3D. For each method, we draw the top-5 retrieved motion sequences, where we rank the results by similarities and give the annotation at the bottom of each motion sequence. We also highlight the semantically similar verbs to the query by the same color. The successful and failed retrieval results are highlighted by the green and red border, respectively. In Fig. 3 , the query contains multiple atomic motions with different semantics, especially the motion "walks" and "holds" are co-occurring. For the TMR, we notice that 1) the successful retrieval result is in the fourth position, which means that the TMR can retrieve semantically similar motions, but with limited precision.2) The first and second returned results only contain only part of the semantics consistent with the query. It may caused by the single-point representation only capturing the simplistic motion semantics ("put" and "walk") and failing to represent the complicated motion sequences. Further aligning these rough representations can significantly disturb the cross-modal alignment and undermine the retrieval performance.For the results of MGSI, we observe that 1) the proposed method successfully retrieves the motion sequences corresponding to the query by accurately identifying these co-occurring motions and learning semantic correlations between the query and candidate from multiple granularities, demonstrating our methods' effectiveness. 2) The second retrieved motion sequence still contains similar The person walk forward and wipe some thing off.A person put down something and slowly walked forward.A person holds both hands out and walks slowly forward, putting their hands down by their sides. A person's walking forward quickly while holding something that they seem to be balancing or positioning.A person jogs forward and then places their hands at their side.A person carefully walks down a slope.A person walks forward and holds a large object in their hands, puts the object down and continues to walk.A person raises their arms, while walking forward.A man walks forwards whilst putting his arms in front of him holding something.A person walks forward and holds a large object in their hands, puts the object down and continues to walk. semantics to the query ("hold", "walks", and "putting down"). It demonstrates that our MGSI can successfully capture fine-grained semantics between the text and motion sequences. 3) Although other results are the failed retrieve results, they still contain these co-occurring motions ("walk" and "hold"), which verifies the effectiveness of the proposed co-occurrence motion mining approach. These observations suggest that through integrating token, instance, and bag-wise semantics interactions, our MGSI can capture both fine-grained and overall semantics, ensuring a comprehensive semantic analysis. 
conclusions and future works: 
 In this paper, we conceptualize the Text-Motion Retrieval (TMR) task as a Multi-Instance Multi-Label (MIML) learning problem, where each motion sequence is viewed as a bag of atomic motions, and the corresponding text as a bag of phrases. To tackle the MIML challenge within TMR, we introduce a novel Multi-Granularity Semantics Interaction (MGSI) approach, which effectively captures and aligns the semantics of text and motion sequences across various levels. Specifically, the MGSI approach decomposes both query and motion sequences into three hierarchical levels: token, instance, and bag. We then utilize graph neural networks to explicitly model and interact with their semantic correlations at these levels, thus capturing the semantics across multiple granularities accurately.To identify and model co-occurring atomic motions, we measure semantic consistency frame-wise, then fuse and interact the accordant motions to refine their representations. Finally, we employ token, instance, and bag-wise semantic interactions to comprehensively align the text and motion sequences. Extensive experiments on two widely-used datasets demonstrate the efficacy of our methods In our future work, we plan to incorporate informative skeleton features to enhance precise atomic motion mining, further facilitating fine-grained semantic interaction between text and motion.  
conventional method: 
 The proposed MGSI this vast repository has become an increasingly critical need. Textmotion retrieval (TMR) [15] is a typical multi-modal retrieval task that aims to retrieve semantically similar motion sequence by the given query text. Following the paradigms of conventional multimodal retrieval, 𝑒.𝑔., text-image and text-video retrieval, numerous TMR researches [4, 12, 14, 15, 21, 24] are raised. Tevet 𝑒𝑡 𝑎𝑙. [21] introduce a MotionCLIP, where a motion auto-encoder is trained not only to reconstruct motion sequences but also to align their latent representations with the corresponding textual and visual representations in the CLIP space [18] . Mathis 𝑒𝑡 𝑎𝑙. [15] propose the task of text-motion retrieval and establish a series of evaluation benchmarks with varying difficulty and introduce a joint synthesis and retrieval framework. Yan 𝑒𝑡 𝑎𝑙. [24] adopt a dual-unimodal transformer encoder to enable a wide range attention in text and motion sequence and introduce a drop triplet loss function to mine the false negative samples.Although existing work has achieved promising retrieval performance, it generally represents both the query text and motion sequences with a single embedding for alignment. However, in text-motion retrieval, the text and motion sequence typically contain multiple instance, 𝑖.𝑒., atomic motions in a motion sequence, verb phrases in a sentence, and include distinct semantics instead of the single samples with unique semantic. As shown in Fig. 1 , the text-motion pair contains three atomic motions, 𝑖.𝑒., "walking", "carrying" and "put down". The conventional methods may fall short in accurately modeling the complicated semantics of these atomic motions due to the simple representation approach. In addition, these multiple atomic motions are usually co-occurring and overlapping with each other. As shown in Fig. 1 , the motion "walking" and "carrying" are co-occurring and overlapping with each other. Solely aligning the single representation of query text and motion sequence may struggle to achieve a accurate cross-modal relation matching in these co-occurring motions, degenerating the retrieval performance. It is necessary to develop a effective method that explicitly represents and accurately aligns multiple atomic motions and corresponding text.In this work, we argue that the text-motion retrieval can be viewed as a classical learning problem, Multi-Instance Multi-Label (MIML) [32] , where sample is defined as a bag of multiple instances and associated with multiple class label. Thus, we regard the textmotion retrieval task as the MIML learning problem, in which the motion sequence is a bag of atomic motions and the query text is a bag of phrases corresponding to atomic motions. To address the MIML problem of TMR, we propose a Multi-Granularity Semantics Interaction (MGSI) approach as shown in right of Fig. 1 . Specifically, in MGSI, we start by decomposing text and motion sequences into three hierarchical levels: token, instance, and bag to represent various granularity semantics. We employ graph neural networks to build a text and motion graph, where the noun phrases, verbs, and sentence (frames, atomic motions, and sequence) are viewed as the token, instance, bag-wise nodes, respectively. Then, we propose a novel co-occurrence motions mining approach that measures the semantics consistency in frame-wise to score the atomic motions. With the consistency score, the co-occurring atomic motions could be identified and fused to generate the co-occurrence features for update the instance nodes. The graph reasoning is applied on the update graph to capture the complex relationships among these components effectively. After that, we introduce a semantic interaction in token, instance, and bag-wise to migrate the semantics correlation between text and motion sequence, achieving a precise cross-modal alignment. Comprehensive evaluations conducted on two widely used benchmark datasets, HumanML3D and KIT-ML, demonstrate that the proposed MGSI surpasses the state-of-the-art methods in a clear margin.The main contributions of our work are summarised as follows:• In this work, we formulate the text-motion retrieval as a Multi-Instance Multi-Label (MIML) learning problem, where text sequences are treated as a bag of verbs and motion sequences as a bag of atomic motion instances. To the best of our knowledge, this is the first attempt to model the textmotion retrieval as MIML problem. • We propose a novel multi-granularity semantics interaction (MGSI) approach to address the MIML problem of TMR, in which we exploit the graph neural networks to decompose the text and motion sequences into token, instance, and bag and perform cross-modal semantics interaction in the corresponding granularity to enable a precise cross-modal alignment.• Extensive experiments on two widely-used benchmark datasets, HumanML3D [4] and KIT-ML [16] , demonstrate that our proposed method surpasses the state-of-the-art, achieving a 23.09% increase in Rsum on HumanML3D and a 21.84% increase on KIT-ML. 
 
 Appendix:  


Decision: There is a consensus to accept this paper. The innovation and rich experimentation were appreciated. 
The authors provided satisfactory responses to the reviewers' comments, providing reasonable justifications for their design choices. 
If the paper is accepted, the authors will have to update the paper as mentioned in the rebuttal.

Reviews: review: paper_summary: paper_summary: This paper focuses on enhancing the representation learning for the text-to-motion retrieval task. The assumption is that a motion consists of several atomic sub-motions with complex semantics and, therefore, the motion is viewed as a bag of sub-motions, and the text as a bag of corresponding phrases. I generally like the main paper idea to represent motions as well as text in a graph-based structure and apply co-occurrence motion fusion, instead of learning global text and motion embeddings.

The main concern I have is the experimental evaluation:

1) The TMR [15] paper defines four evaluation protocols (All, All with threshold, Dissimilar subset, Small batches). However, these four protocols are not ever considered in this paper. Thus, it is not clear which protocol was really used for experimenting and whether the accuracy of the proposed approach can be considered better than TMR (it can be better only on some protocols)...

2) The proposed approach is better on HumanML3D and only slightly better on KIT-ML, when comparing individual R@x values in Table 1 and Table 2.

3) The authors mention that: "To compare the results fairly, we adopt the open-source code of DTL to train the model in our splits and report the results.", so they have re-evaluated the results for the DTL approach. However, how have they obtained the results for MoT [12]??? The MoT paper [12] (as the first text-to-motion retrieval approach) does not provide direct results for the motion-to-text scenario, so the authors have probably extracted the implementation of MoT (motion transformer) from the provided source codes and integrated it into the evaluation pipeline, haven't they? Btw, the presented results of MoT are lower on the text-to-motion scenario compared to the results reported in [12]. This is not commented on in the paper.

4) In Table 4, most of the best results (all on the "Motion->Text" scenario) belong to the baseline, i.e., w/o downsample and not to MGSI as wrongly highlighted by bold font.

5) The construction of the graph using a "brute-force" approach (i.e., testing all possible sliding windows) in Section 4.2 should be described and motivated in more detail. Since this approach is computationally expensive, I ask for adding some quantitative results -- at least report the total pre-processing/training times.


Text comments:

At the beginning of Introduction, I miss a brief introduction about what motion data are, what are their applications, and a broader context of the retrieval operation in the context of motion data, as surveyed, for example, in:

Sedmidubsky, J., Elias, P., Budikova, P., Zezula, P.: Content-Based Management of Human Motion Data: Survey and Challenges. IEEE Access, 9, 2021.
https://ieeexplore.ieee.org/document/9416451

At the same time, Introduction can be shortened since the main paper idea "...motion sequence is a bag of atomic motions and the query text is a bag of phrases corresponding to atomic motions..." is repeated several times in the introduction. I do not also like that the content of the first paragraph of Introduction overlaps quite a lot with the content of Section 2.1.

In Section 2.2, the text content starting from the sentence "In this work, ..." is already known and again repeated; however, the principal difference between the proposed MIML approach and existing MIML methods is not explained. Similarly, the content of Section 3 contains a lot of information that is already known. This unnecessarily prolongs the text and distracts from important things. However, I would appreciate a better explanation of many symbols used to define the graph G_t (in Section 3) and also the symbols used in Section 4.2. I also ask for a more detailed description of the training procedure in Section 4.6, which is not very clear. Why do not use terms "t_i"/"m_i" instead of undefined terms "x_i" and "y_i" in Equation 10?


Other comments:
* in the paper title: "... for Text-motion Retrieval" --> "... for Text-Motion Retrieval"
* notation is sometimes confusing, e.g., the definition of the motion clips M_c
* "..., the instance nodes m^1_a and m^1_a..." --> "..., the instance nodes m^1_a and m^2_a..."
* quite a lot of misspellings/errors, e.g., "toke-wise" --> "token-wise", "a effective method" --> "an effective method", "the the model" --> "the model", "multi layer perception" --> "multi layer perceptron", "in the retrieve related tasks" --> "in the retrieval-related tasks", "efficacy of our methods" --> "efficacy of our methods."
strength_weakness: strength_weakness: The main paper idea is interesting and new in the context of text-to-motion retrieval, i.e., representing motions and text as bags of atomic motions/phrases and applying co-occurrence motion fusion, instead of learning global text and motion embeddings.
limitations: limitations: Experimental evaluation is principally well-designed, but there are some misunderstandings. The text description, as well as the formal notation, should be enhanced. See the Summary part for more details.
paper_summary: paper_summary: This paper aims to address the problem of capturing complex semantics of motion in text-motion retrieval task by treating it as a Multi-instance Multi-label (MIML) learning problem. The authors introduce a Graph Neural Network (GNN)-based Multi-Granularity Semantics Interaction (MGSI) approach which constructs both a text graph and a motion graph. The text graph includes noun phrases, verbs, and bag-wise node, while the motion graph consists of frames, atomic motions, and bag-wise node. Interactions between the corresponding nodes of the two graphs capture multi-level semantic information. Experiments conducted on two commonly used datasets have achieved SOTA performance.
strength_weakness: strength_weakness: 1. **Motivation:** The motivation behind this paper is quite clear.
2. **Experiments:** This work has conducted adequate experiments to validate the effectiveness of the proposed method.
limitations: limitations: 1. **Novelty:** Many researchs in retrieval-related tasks have shown that more interactions lead to better performance but also result in higher computational costs. The way the authors improves the TMR task may lack novelty. Additionally, the method proposed requires constructing a motion graph based on the text query, which means that motion sequences cannot be pre-processed, leading to potentially high latency. A comparison of latency should be provided. Here are some related references:
     - Khattab, O., & Zaharia, M. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, SIGIR 2020.
     - Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, Jason Weston. Poly-encoders：Architectures and Pre training Strategies for Fast and Accurate Multi sentence Scoring, ICLR 2020.

2. **Writing:** There are a few minor points that are unclear.  
      **a.** In the explanation of Equation 1, the author mentions the set ${𝑚_𝑎^1,…,𝑚_𝑎^𝑘}$ but does not clarify what 'k' represents. Does the 'k' refer to the number of noun phrases?  
      **b.** In Table 4, removing the downsample strategy resulted in a worse performance for text → motion, which contradicts the statement in the paper that "when removing the downsample strategy, the retrieval results are further boosted but limited."
paper_summary: paper_summary: This paper focus on the challenge that co-occurring and coupling of atomic actions with complex semantics is hard to be represented by single
embeddings in the text-motion retrieval task. It formulates TMR as a Multi-Instance Multi-Label learning problem and  propose a novel Multi-Granularity Semantics Interaction (MGSI) approach, which effectively captures and aligns the semantics of text and motion sequences across various levels.

 Overall,  this is a valuable work in my opinions. It offers a new perspective to consider the text-motion retrieval problem from the viewpoint of MIML. The proposed framework appears to be coherent and logically structured. It aligned semantics from the perspectives of tokens, instances, and bags respectively. The experiment also appears to be sufficient and convincing
strength_weakness: strength_weakness: 1.  This paper formulates TMR as a Multi-Instance Multi-Label(MIML) learning problem which is the first attempt in this field.
2.  It proposes a novel Multi-Granularity Semantics Interaction (MGSI) approach to explicitly model the semantics in 3 levels and estabilished their semantic consistency.
3. The obvious improvement on experiment results shows their efficacy.
limitations: limitations: 1. The result in Figure 4  seems to show the model is sensitive on \lambda. Is the value of this hyperparameter sensitive to the dataset? Do different datasets require independent settings? At least a discussion is needed.
paper_summary: paper_summary: The authors tackle text-motion retrieval using the framework of multi-instance multi-label (MIML) learning. Specifically, they propose to decompose both texts and motions into their building blocks at three different granularity levels (bag, instance, token). Successively, they perform some filtering of the constructed graphs to remove redundant information and fuse instance nodes if they contain too similar token nodes. They then aggregate information at different granularity levels using graph neural networks and enforce three different losses for the information extracted at each level. They perform experiments on two widely employed datasets for text-motion retrieval.
strength_weakness: strength_weakness: - The paper tackles the important problem of separating fine-grained semantics of motions that, differently from images or videos, contain much higher inter-class confusion. The idea of employing a multi-instance learning framework, in this context, is sound.
- The results on both KIT and HumanML3D datasets show the effectiveness of the proposed approach with respect to the baselines, where a consistent improvement margin is obtained.
- The final ablation study validates the components of the proposed architecture.
limitations: limitations: - One of the claims of this paper is that global features approaches used by the baselines are not sufficient to solve this task. This is further remarked in the ablation study in Table 3, where only the bag (global) feature matching obtains a performance comparable with the early baselines. However, even recent approaches (TMR) employ global features but obtain much better results. In this sense, it is strange that the performance of solely the bag components achieves such low results. I would have expected the bag features to be comparable with TMR. This could signify that the global features from the proposed architecture carry too little information compared with other global-based descriptors.
- It is strange to frame the problem using a multi-label framework. While it is clear that it could be multi-instance, it is strange to treat it as multi-label, given that labels are not explicit and only appear in the final natural language sentences. In this sense, it may not be a multi-label task from the standard perspective, which should be better stated in the text.
- The clip selection procedure, as shown in Table 5, seems very important, as the performance triples with respect to the random selection. However, the selection is performed using a pre-trained text-motion model. This means that the proposed method may be highly dependent on one of the baselines (TEMOS in this case). I would expect the authors to better comment on this (e.g., did they try other more performant methods (e.g., TMR [15]) or other selection criteria?)
- Why did the authors use a transformer for the token-wise interaction while instead a simple aggregation of the cosine similarities for bag and instance interactions? Did they also try a simpler aggregation for token-wise interaction?
- There are wrong bolds in Table 4 (the bold ones are not the best values). This means that, in some cases, the best results are not the ones where all the modules collaborate to reach the best performance. In the light of this, a more careful discussion in the ablation studies is needed.

Minor issues:
- I believe S_{token} in Eq. 7 is obtained through a sigmoid and not a softmax from the output of the MLP (I assume the MLP output is 1-dimensional). If so, I also assume cross-entropy in Eq. 11 is a binary cross-entropy loss function. If so, please change softmax to sigmoid in Eqs. 7 and specify that is a binary CE in Eq. 11.
- The paragraph title “Instance-wise Interaction” in Section 4.5 should be on a new line.