Cross-Task Knowledge Transfer for Semi-supervised Joint 3D Grounding and Captioning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 06:47:41

Abstract: 3D visual grounding is a fundamental but important task in multimedia understanding, which aims to locate a specific object in a complicated 3D scene semantically according to a text description. However, this task requires a large number of annotations of labeled text-object pairs for training, and the scarcity of annotated data has been a key obstacle in this task. To this end, this paper makes the first attempt to introduce and address a new semi-supervised setting, where only a few text-object labels are provided during training. Considering most scene data has no annotation, we explore a new solution for unlabeled 3D grounding by additionally training and transferring samples from a correlated task, i.e., 3D captioning. Our main insight is that 3D grounding and captioning are complementary and can be iteratively trained with unlabeled data to provide object and text contexts for each other with pseudo-label learning. Specifically, we propose a novel 3D Cross-Task Teacher-Student Framework (3D-CTTSF) for joint 3D grounding and captioning in the semi-supervised setting, where each branch contains parallel grounding and captioning modules. We first pre-train the two modules of the teacher branch with the limited labeled data for warm-up. Then, we train the student branch to mimic the ability of the teacher model and iteratively update both branches with the unlabeled data. In particular, we transfer the learned knowledge between the grounding and captioning modules across two branches to generate and refine the pseudo labels of unlabeled data for providing reliable supervision. To further improve the pseudo-label quality, we design a cross-task pseudo-label generation scheme, filtering low-quality pseudo-labels at the detection, captioning, and grounding levels, respectively. Experimental results on various datasets show competitive performances in both tasks compared to previous fully- and weakly-supervised methods, demonstrating the proposed 3D-CTTSF can serve as an effective solution to overcome the data scarcity issue.

Main paper: 
 introduction: 
 3D visual grounding (3DVG) [1, 8] is a fundamental yet important task in multimedia understanding, which has recently received increasing attention due to its wide range of applications, such as robotic navigation and Augmented Reality / Virtual Reality systems. As shown in Figure 1 (a), 3DVG aims to locate the target object in a 3D point cloud scene based on a given free-form query text description. Since 3D scenes generally contain complicated background objects and spatial relationships, this task needs to not only model the complex multi-modal interactions among features of the point cloud and language, but also capture complicated context information for their semantic reasoning.Existing methods for the 3DVG task can be generally categorized into two types based on task settings: fully-supervised [1, 7-9, 20, 22, 45, 49, 51, 52] and weakly-supervised methods [44, 48] . As shown in Figure 1 (b), under the fully-supervised setting, a large number of bounding boxes for numerous objects in the 3D point cloud scenes, along with their descriptive texts, are required to be annotated as labels to provide reliable supervision. For example, the widely used ScanRefer dataset [8] contains 806 scenes with 51,583 matched object-text labels. However, manually annotating these object-text pairs is very time-consuming and labor-intensive. To alleviate this problem, the weakly-supervised methods [44, 48] are proposed to only use text labels for grounding without relying on any object bounding box annotations. Although they slightly reduce the annotation cost, tens of thousands of language descriptions still require significant manual effort for annotation.Based on the above observation, in this paper, we focus on how to learn a 3D visual grounding model with as few annotations as possible. In particular, we introduce a new semi-supervised setting for the 3DVG task, where only a small number of object-text pair labels (about 10%) in 3D scenes need to be manually annotated while the vast majority of scenes have no manual annotations. Since most scene data lack annotations, we explore the idea of cross-task assistance and knowledge transfer from a related task, i.e., the 3D captioning task that aims to generate a contextual sentence description corresponding to a given object within the 3D scene. The motivation is illustrated in Figure 1 (c) , where we observe that both tasks are complementary to each other as grounding can provide detected objects for feeding captioning while captioning can generate reliable sentences to guide the grounding process. However, it is non-trivial to directly build a collaborative optimization scheme for these two tasks: (1) Efficient model design. Although directly employing a pre-trained 3D captioning model to provide guidance for unlabeled 3D grounding does work, this defeats our purpose of only using a small number of annotations for training, and introduces large memory costs. How to efficiently and effectively train both 3D grounding and captioning models from scratch with annotation-limited grounding data is crucial. (2) Framework for addressing the semi-supervised setting. In the semi-supervised setting, the limited annotations cannot provide strong supervision to generalize our grounding and captioning models. How to design a framework that can both learn the capability of addressing the task from the limited annotations and fine-tune the models for better generalization with the large amount of unlabeled data is also essential. (3) Cross-task knowledge transfer. Instead of individually training the grounding and captioning models, it is important to have them cooperate to provide additional contexts for each other during the learning. Especially when learning with unlabeled data, how to explore their task-specific knowledge to provide rectification on pseudo-labels for each other for better updating the models is challenging.To address the above issues, we propose a novel approach, called 3D Cross-Task Teacher-Student Framework (3D-CTTSF), to jointly learn both 3D grounding and captioning tasks in a single backbone model with two different task-specific heads. To handle the challenging semi-supervised learning, 3D-CTTSF designs two branches sharing the same joint backbone to build a teacher-student framework. Specifically, 3D-CTTSF first pre-trains the two task-specific heads of the teacher branch with limited labeled data to acquire a certain level of proficiency in tackling the tasks for warm-up. Then, it trains the student branch to mimic the ability of the teacher model and utilizes the pseudo-labels of unlabeled data predicted by the teacher model for updating the student branch. To make two task-specific heads cooperate to provide learning guidance for each other, we transfer the learned knowledge between the grounding and captioning modules across two branches. This transfer helps rectify and refine the pseudo-labels of unlabeled data for providing reliable supervision. To further improve the pseudo-label quality, we design a cross-task pseudo-label generation scheme, filtering out low-quality pseudo-labels at the detection, captioning, and grounding levels, respectively. Through pseudo-label training and knowledge transfer, we address not only the semi-supervised 3D visual grounding task but also the semi-supervised 3D dense captioning task. During the inference, we directly feed the data into the student model for prediction.Our main contributions can be summarized as follows:• To the best of our knowledge, we are the first to address the 3D visual grounding task in a challenging semi-supervised setting. To handle the limited annotation, we propose a novel paradigm that explores cross-task assistance from a complementary task of 3D captioning. • To tackle semi-supervised learning, we develop a teacherstudent framework based on a joint grounding and captioning backbone to train with labeled data and update with pseudo-labels of the unlabeled data. Cross-task knowledge transfer and pseudo-label filtering modules are further developed to aid learning from unannotated scenes.• Extensive experiments are conducted on various datasets of both grounding and captioning. Results show that our proposed 3D-CTTSF bridges the gap between semi-supervised learning and fully-supervised methods. 
related work: 
 3D Visual Grounding. Existing 3D visual grounding methods can be generally classified into two-stage and one-stage approaches based on their model designs. Two-stage methods [1, 3, 7, 8, 12, 16, 18, 20, 21, 34, 49, 51, 52] first utilize pre-trained 3D detectors [17, 33, 54] or segmentors [15] to generate a large number of candidate proposals, and then fuse proposal features with textual features to select the object that best matches the given text. One-stage methods [22, 29, 45] directly guide object detection using textual cues, making it easier to locate the object related to the query text. However, almost all of them are developed in the fully-supervised setting where a large number of object-text annotations are required. Although a few weakly-supervised methods [44, 48] are proposed to only use text labels for grounding without relying on any object bounding box annotations, a large number of language descriptions still need to be annotated. Different from them, we are the first to attempt to solve this task with a semi-supervised approach. Although D3net [9] claims applicability in a semi-supervised setting, they still utilize all ScanRefer data as labeled data and only employ some unannotated objects from Scannet as unlabeled data. This setting is much simpler than ours but has not demonstrated superior performance. 3D Dense Captioning. Another focal point in 3D multi-modal scene understanding is 3D dense captioning [11] . This task aims to generate natural language descriptions for each object within a scene. Similar to visual grounding, methodologies in 3D dense captioning can be classified into two categories: two-stage and onestage methods. Two-stage method [11, 24, 43, 50] follow "detectthen-describe" pipeline. They first detect many proposals and then generate captions for each of them. The one stage method [10] directly creates object queries from the scene. The query then be decoded for detection and captioning.Semi-Supervised Learning. Semi-Supervised Learning (SSL) has made significant progress in various tasks, such as image classification [5, 6, 19, 36] , object detection [23, 26, 27, 30, 37, 53] and referring expression comprehension [38] . The methods of semisupervised learning can be roughly divided into consistency based methods [5, 6, 23, 36] and pseudo-label based methods [2, 26, 27, 30] . Consistency-based methods utilize consistency regularization, which forces both the teacher and student models to produce the same predictions from differently augmented inputs. On the other hand, pseudo-label based methods supervise the training of the student model based on pseudo-labels generated by a reliable teacher model. Many techniques have been proven effective in semi-supervised learning and have thus been widely applied, such as EMA [39] update for teacher, asymmetric data augmentation [35, 46] , and pseudo-label filtering [36] . 
the proposed method 3.1 overview: 
 Task definition. Semi-supervised 3D grounding and captioning aim to utilize a single pipeline to jointly locate the text-related object and generate the object-guided sentence with only a few annotations. To be specific, the given semi-supervised dataset includes a small set of labeled data 𝐷 𝑙 = {P 𝑙 𝑖 , L 𝑙 𝑖 , B 𝑙 𝑖 } 𝑁 𝑠 𝑖=1 and a large set of unlabeled data 𝐷 𝑢 = {P 𝑢 𝑖 } 𝑁 𝑢 𝑖=1 , where P, L, B denote point cloud of a 3D scene, language description and localization of objects, respectively. 𝑁 𝑠 and 𝑁 𝑢 are the numbers of labeled data and unlabeled data. In practice, they often meet 𝑁 𝑠 ≪ 𝑁 𝑢 . Overall pipeline. To address the joint tasks in the challenging semi-supervised setting, we propose a novel 3D Cross-Task Teacher-Student Framework (3D-CTTSF). Both the teacher and student branches share a similar architecture with a detection module and parallel grounding and captioning modules. As shown in Figure 2 , the overall training process consists of two stages. In the first stage, we pre-train the grounding and captioning modules of the teacher model with a few annotated data for warm-up. By doing so, we obtain a teacher model with a certain capability to perform both grounding and captioning tasks. Then, we copy the parameters of the teacher model to the student model in preparation for the next stage. In the second stage, we train the student branches with both labeled data and unlabeled data. Unlabeled data is supervised using pseudo-labels, while labeled data is supervised using the ground truth. Specifically, we utilize the teacher model to generate the pseudo object proposals and corresponding pseudo textual descriptions for the unlabeled 3D scenes. To improve the quality of pseudo-labels, we introduce a cross-task pseudo-label generation scheme, filtering out low-quality pseudo-labels at the detection, captioning, and grounding levels, respectively. Then, we apply these pseudo-labels to train the student model while the teacher model also updates its parameters from the student model using exponential moving average (EMA) [39] strategy. During the inference, we directly feed the test data into the student model for prediction.Backbone Model. We follow previous work [7] to design our basic joint grounding and captioning backbone model, which consists of three main modules: an object detection module O, a grounding module G, and a captioning module C. In the object detection module, the input 3D scene point cloud is processed through a Votenet [33] to obtain initial object proposals. For the captioning module, it selects the proposal with the maximum Intersection over Union (IoU) value with the input bbox to generate a caption in training, and directly generates captions for all proposals in testing. For the grounding module, it predicts confidence scores for all proposals based on the input text. Design of IoU estimation module. To adapt the backbone to semi-supervised setting, we require the model to handle the pseudolabels of unlabeled data. To evaluate the quality of pseudo-labels, we design an IoU estimation module upon the detection module to predict the IoU values between each proposal and the ground truth bounding box. To fully capture the geometric information of proposals, inspired by BRNet [13] , we first sample two representative points in each of the six directions (up, down, left, right, front, back) around the given proposal. These 12 number of representative points are evenly distributed along the axes from the bounding box center to the center of each face. By aggregating the contextual features of seed points around these representative points, we obtain and fuse the features of all representative points to predict the IoU value of the proposal.Semi-supervised SettingAfter introducing the basic backbone of joint tasks, we then illustrate how we build the detailed framework for addressing the challenging semi-supervised setting.How to tackle the semi-supervised learning? In the semisupervised setting, we only have access to very few paired objectquery samples while the others are all unlabeled. A general way to address this setting is to utilize a teacher-student framework. Specifically, the teacher model is initially trained on a limited number of annotated samples for several epochs to acquire a certain level of proficiency in the given task. Subsequently, the learned teacher model is utilized to assign pseudo-labels to the other unlabeled data.The unlabeled data, now augmented with pseudo-labels, can be further utilized alongside labeled data to train the student model. Design of the teacher-student framework. Our teacher-student framework consists of a teacher model and a student model, which share the same architecture of the joint task backbone [O, C, G]. For clarity, variables and modules associated with teacher and student models are denoted with superscripts "𝑡" and "𝑠", respectively. The first stage, also called the warm-up stage, is to pre-train the teacher model with a few annotated samples. Specifically, we train the teacher model using annotated dataset 𝐷 𝑙 = {P 𝑙 𝑖 , L 𝑙 𝑖 , B 𝑙 𝑖 } 𝑁 𝑠 𝑖=1 ,EQUATION)where 𝛾 is the learning rate and 𝑗 is the training step index. After training in warm-up stage for 𝐸 1 epochs, we copy the parameters of the teacher model to the student model and start the training for the semi-supervised stage for 𝐸 2 epochs. In the second stage, we update the weights of the student model by using pseudo-label learning with unannotated samples. To be specific, the unlabeled dataset 𝐷 𝑢 is first fed into the teacher model to generate the pseudo-labels, and then the pseudo-labels are utilized to train the student model. In particular, we employ a training strategy of strong data augmentation for the student model and weak data augmentation for the teacher model. This strategy enables the teacher model to have a performance advantage over the student model, allowing the student to learn from the teacher. Specifically, for the input scene point cloud of the teacher model, we only apply point cloud resampling as weak data augmentation. For the student model, in addition to resampling, we employ a combination of random flips along the y-axis, random translations, random size scaling, and random slight rotations as strong data augmentation. Note that random flipping may change the relative positions of objects, so corresponding modifications to the textual descriptions are required. We did not adopt flipping along the x-axis due to the relatively complex expression of front-back relationships in natural language. For flipping along the y-axis, we can simply swap the words "left" and "right" in the textual description. After asymmetric data augmentation, the input point clouds of the teacher and student models are denoted as P 𝑡 and P 𝑠 , respectively.Next, P 𝑡 will be input into the teacher model for generating corresponding pseudo-labels, and the obtained pseudo-labels will be used for training the student model. The teacher model is also updated through an EMA strategy illustrated in the latter section. To obtain high-quality pseudo-labels for better training the whole teacher-student framework, we design a cross-task pseudo label generation for supervising different modules in the next section.Due to the limited number of labeled data and the complexity of the joint task, the performance of the teacher model does not guarantee the generation of high-quality pseudo-labels. However, low-quality pseudo-labels will reduce the learning efficiency and effectiveness of the student model. This necessitates a filtering process of the pseudo-labels generated by the teacher model to provide true-positive labels for learning student model. We consider improving the quality of pseudo-labels across tasks in three parts: detection module, captioning module, and grounding module. Pseudo label of detection module. As shown in Figure 2 EQUATIONThe features of the objects are represented as P 𝑡,0 ∈ R 𝐾 0 ×𝐹 0 . Then we use a detection-based filter F 𝑑 to filter out bounding boxes with poor quality based on 𝑜 𝑡 ,c 𝑡 and 𝑖 𝑡 . The filtering process involves three threshold judgments: using 𝑜 𝑡 > 𝜏 obj to filter out proposals that may not be foreground objects; using max(c t ) > 𝜏 cls to filter out proposals that are difficult to determine the category of the object; using 𝑖 𝑡 > 𝜏 iou to filter out proposals that do not accurately locate the object. We sort the retained objects based on the global scores 𝑠 ×max(c) ×𝑖 and select the top 𝐾 1 objects with the highest product as the high-quality pseudo-labels {B 𝑖 𝑡,1 } 𝐾 1 𝑖=1 . The process of selecting 𝐾 1 objects' features P 𝑡,1 and {B 𝑖 𝑡,1 } 𝐾 1 𝑖=1 is:EQUATIONPseudo label of captioning module. The obtained high-quality object features P 𝑡,1 are then fed into the captioning module C 𝑡 of the teacher model to generate pseudo-captions for these 𝐾 1 objects {B 𝑖 𝑡,1 } 𝐾 1 𝑖=1 . The generated captions are denoted as {L 𝑖 𝑡,1 } 𝐾 1 𝑖=1 . However, many of these generated pseudo-captions are still of poor quality and cannot be used for cross-task training. Therefore, we adopt a simple strategy to filter out low-quality texts and only choose the high-quality text-object pair for later training. First, if the length of the pseudo-captions len(L 𝑖 𝑡,1 ) < 𝜏 cap , the caption is likely to be several meaningless words and we filter is out. If the pseudo-captions do not contain any words or phrases describing the target object category, they should also be filtered out. When the number of unfiltered pseudo-captions reaches the preset number 𝐾 2 , the generation process ends. If the remaining number of pseudocaptions is less than 𝐾 2 , they are padded with empty texts. Note that these empty texts will not be used for the training of the student model. After this filtering stage, we obtain 𝐾 2 high-quality pseudocaptions {L 𝑖 𝑡,2 } 𝐾 2 𝑖=1 and their corresponding objects {B 𝑖 𝑡,2 } 𝐾 2 𝑖=1 . Pseudo label of grounding module. Previously obtained 𝐾 2 pseudo-captions may not necessarily provide sufficient information to locate the target object, or may not pinpoint a unique object. Therefore, we apply the grounding module G 𝑡 of the teacher model to further filter out pseudo-captions that cannot be accurately localized by the teacher model's grounding module. Specifically, we first input the 𝐾 2 pseudo-captions {L 𝑖 𝑡,2 } 𝐾 2 𝑖=1 and all unmatched 𝐾 1 objects detected by the detection module O 𝑡 into the grounding module G 𝑡 for matching, resulting in a set of confidence scores. The confidence score s ∈ R 𝐾 1 for a certain pseudo-caption L 𝑖 𝑡,2 represents its matching score with all 𝐾 1 objects. Then, we utilize the matched objects B 𝑖 𝑡,2 of 𝐾 2 pseudo-captions as guidance, and extract scores corresponding to the objects B 𝑖 𝑡,2 associated with the pseudo-description text, which reflects the ability of this pseudocaption to localize the correct object. By comparing the scores, we sort and select the top 𝐾 3 pseudo-captions {L 𝑖 𝑡,3 } 𝐾 3 𝑖=1 with their corresponding objects {B 𝑖 𝑡,3 } 𝐾 3 𝑖=1 . Due to the different data augmentations applied to the input point clouds of the student and teacher, further pre-processing of the pseudo-labels is required before they can be utilized. Let T represent the combination of transformations applied to the student's input point clouds. Consequently, the pseudo-labels 𝑏 generated by the teacher for object positions must undergo the same transformations 𝑏 𝑇 = T (𝑏). Furthermore, if the transformation T involves flipping along the y-axis, the pseudo-text descriptions generated by the teacher should also be modified accordingly, with the words "left" and "right" being interchanged. Following this processing, the student model can leverage these pseudo-labels for training.Since the 3D grounding and captioning are complementary, simultaneously training these two tasks allows us to perform cross-task knowledge transfer: during the semi-supervised training stage, we transfer the learned knowledge between the two tasks with unlabeled data. Specifically, the predicted text obtained by the teacher model's captioning module C 𝑡 is first converted into tokens and encoded using GloVE [32] . Then, after pseudo-label filtering, this textual output serves as description text input for the student model's grounding module G 𝑠 . In this manner, we transfer the knowledge learned by the captioning module of the teacher model to the grounding task on unlabeled data. Conversely, we employ the grounding module G 𝑡 of the teacher model to determine whether a piece of text can accurately locate the target object. This enables the selection of distinct object-text pairs for training the captioning module C 𝑠 of the student model to generate discriminative captions, thereby transferring the knowledge learned by grounding module of teacher model to the captioning task on unlabeled data.To be specific, we utilize the pseudo-labels of detection module {B 𝑖 𝑡,1 } 𝐾 1 𝑖=1 and pseudo-label of object-text pair {L 𝑖 𝑡,3 } 𝐾 3 𝑖=1 , {B 𝑖 𝑡,3 } 𝐾 3obtained from the teacher model to train the student model. The scene point cloud P 𝑠 is first inputted into the detection module D 𝑠 of the student model, yielding 𝐾 0 bounding boxes B 𝑠 of objects and their corresponding features P 𝑠 via:EQUATIONThese features P 𝑠 are subsequently fed into both the grounding module G 𝑠 and captioning module C 𝑠 to accomplish the two tasks.For the grounding task, we query the object in B 𝑠 that matches a particular textual pseudo-label {L 𝑖 𝑡,3 } 𝐾 3 𝑖=1 best:EQUATIONwhere s 𝑠,𝑖 ∈ R 𝐾 0 represents the matching score between the given text and the 𝐾 0 objects B 𝑠 . We supervise it using corresponding object pseudo-labels {B 𝑖 𝑡,3 } 𝐾 3 𝑖=1 . For the captioning task, we identify the object in B 𝑠 with the highest IoU with a given object pseudolabel {B 𝑖 𝑡,3 } 𝐾 3 𝑖=1 and generate a textual description for it:EQUATION)We supervise it with the pseudo-label {L 𝑖 𝑡,3 } 𝐾 3 𝑖=1 associated with that object. Additionally, we also employ detection pseudo-labels {B 𝑖 𝑡,1 } 𝐾 1 𝑖=1 to supervise the results of object detection B 𝑠 in the student branch. The overall loss function can be expressed as:EQUATIONwhere L 𝑙 , L 𝑢 are the loss functions for labeled and unlabeled data (illustrated in Sec. EQUATION)where 𝛼 is an EMA smoothing coefficient, 𝛾 is the learning rate and 𝑗 is the training step index. The whole semi-supervised training algorithmic procedure pseudo-code is shown in Algorithm 1.Labeled data loss. We utilize ground truth to compute the labeled loss L 𝑙 . Given the integration of both visual grounding and dense captioning tasks, a combination of multiple loss functions is required to supervise the training process. The first component of the loss function is the detection loss 𝐿 𝑙,det , which consists of the losses for objectness score, object category prediction, IoU prediction, and bounding box prediction, denoted as 𝐿 𝑙,obj , 𝐿 𝑙,sem , 𝐿 𝑙,iou , and 𝐿 𝑙,box :EQUATIONThe second component of the loss function is the grounding task loss 𝐿 𝑙,grd . Similar to ScanRefer [8] , it is obtained by computing the loss between the confidence score and the one-hot target label obtained from the ground truth. The third component is the captioning task loss 𝐿 𝑙,cap , representing the cross-entropy loss for token prediction [41, 47] . The overall loss function for labeled data is the weighted sum of these three components:EQUATIONUnlabeled data loss. We compute unlabeled data loss L 𝑢 using pseudo-labels generated by the teacher model. The computation method is generally similar to that of labeled data, but for unlabeled data, we no longer supervise the prediction of IoU and objectness score [42] , only using them for pseudo-label filtering:EQUATION 
experiments 4.1 datasets and evaluation metric: 
 For the 3D grounding task, we conducted experiments on widely used ScanRefer dataset [8] and Nr3D dataset [1] . ScanRefer contains a total of 51,583 textual descriptions corresponding to the objects provided in 806 scanned scenes from the ScanNet [14] dataset. We utilize the metric Acc@kIoU, where "k" represents the minimum threshold for the IoU between the predicted bounding box and the ground truth. Following previous works [8, 49] , we set "k" to 0.25 and 0.5 for our experiments. To provide a comprehensive analysis, we presented the results separately for the "unique" and "multiple" subsets. The "unique" subset refers to scenes where there is only one object of the same category as the target object, while the "multiple" subset includes scenes with multiple objects of the same category. Nr3D [1] provides 41.5K textual descriptions for scenes in ScanNet. We evaluate the effectiveness of our method on NR3D using the same metrics. For the 3D captioning task, we followed the approach of Scan2Cap [11] to handle the description text of the ScanRefer dataset. Texts exceeding 30 tokens were truncated to 30 tokens, with [SOS] and [EOS] tokens added at the beginning and end, respectively. To evaluate the performance of the method on captioning tasks, we adopted the common used m@kIoU as metric, where m represents captioning metrics CiDEr [40] , BLEU-4 [31] , METEOR [4] , and ROUGE-L [25] . These metrics jointly measure the quality of both object detection and text generation.We utilized four NVIDIA RTX3090 GPUs for training. During the warm-up and pseudo-label learning stages, we trained for 200 epochs (i.e., 𝐸1, 𝐸2 = 200) each with a batch size of 8. For each stage, we used an AdamW [28] optimizer with a cosine learning rate decay strategy. The initial learning rate for warm-up stage was set to 5e-4, while for the pseudo-label learning stage, it was set to 2.5e-4. In our experiments, the thresholds 𝜏 obj , 𝜏 cls , 𝜏 iou , and 𝜏 cap used in the pseudo-label filtering stage were set to 0.5, 0.5, 0.15, and 5 respectively. The remaining numbers of pseudo-labels after filtering for each stage 𝐾 1 , 𝐾 2 , and 𝐾 3 , were set to 128, 16, and 8 respectively. As for the loss function, the coefficients 𝜆 1 , 𝜆 1 , we present a comparison of our method with previous fully supervised and weakly supervised approaches for the 3D grounding task on the ScanRefer dataset.From the table, it can be observed that with only 10% of labeled data, our method achieves better results than many fully supervised methods that utilize all labeled data, although slightly inferior to some state-of-the-art methods. This demonstrates the effectiveness of our approach. The table also showcases two weakly supervised methods, which utilize all textual annotations but lack corresponding bounding boxes. Compared to them, our approach solely utilizes a small amount of fully labeled data along with a large amount of unlabeled scene point cloud data. Furthermore, the 3D-VLA [48] method heavily relies on pre-trained models in the 2D domain. Therefore, directly comparing these weakly-supervised methods to us is not feasible. However, we still outperforms their performance a lot. To fairly compare the existing methods in the same semi-supervised setting, we also re-implement previous approaches by using only 10% labeled data. As shown in Table 2 , our method significantly outperforms all previous methods when using the same amount of labeled data. This clearly validates the ability of our approach to acquire knowledge for 3D visual grounding from unannotated scenes through pseudo-label learning and knowledge Table 3 : Results of 3D captioning task on Scan2Cap. Method label C@0.5 B-4@0.5 M@0.5 R@0. 3D captioning results.We also compare the performance of our generated captions with several captioning methods in Table 3 . It can be observed that our approach with only 10% labeled data achieves comparable results to many methods utilizing 100% labeled data. This indicates that our method also gains the ability to learn captioning task from unlabeled data through knowledge transfer.Ablation study on varying proportions of labeled data. We conducted experiments under various proportions of labeled data, as shown in Figure 4 . The results indicate that our approach is capable of learning useful information from a small amount of labeled data (e.g., 3%). As the proportion of labeled data increases, our method is able to generate better pseudo-labels, thereby enhancing the final experimental outcomes. Furthermore, comparisons with the baseline method 3DJCG under different annotation proportions reveal significant and consistent enhancements across different proportions of labeled data, demonstrating the ability of our method in learning from unlabeled data.Ablation study on pseudo-label filtering step. We propose three pseudo-label filtering schemes based on detection, captioning, and grounding. To assess the effectiveness of these filtering schemes, we conducted ablation study and presented the results in the detection, captioning, and grounding levels all contribute to improving the results of semi-supervised learning.Ablation study on the filtering thresholds. We conducted further ablation studies on the thresholds used in the pseudo-label filtering step. The experimental results for the 𝜏 obj , 𝜏 cls , and 𝜏 iou are shown in Table 5 . The table shows that both excessively large or small thresholds can compromise the effectiveness of the method. Excessively large thresholds may result in an insufficient number of pseudo-labels, thus preventing the student model from receiving adequate training. Conversely, excessively small thresholds may lead to the utilization of some low-quality pseudo-labels for training, thereby compromising the training results. More ablation study can be found in our Supplementary files.In Figure 3 , we present visualizations of some output results obtained using our approach with 10% annotated data. It can be observed that our method is capable of performing both 3D visual grounding and dense captioning tasks with a minimal amount of annotated data. It is noteworthy that the generated captions uniquely  
conclusion: 
 In this paper, we propose to address an important yet challenging task, i.e., semi-supervised 3D grounding. Since most scene data has no annotation, we explore additionally training a 3D captioning module to assist the grounding learning. To this end, we develop a joint 3D grounding and captioning backbone with a teacherstudent framework design to tackle this semi-supervised learning. Cross-task pseudo-label generation and knowledge transfer strategies are further introduced to improve pseudo-label learning and updating. Extensive experiments demonstrate the proposed framework bridges the gap between semi-supervised learning and fully-supervised learning on various datasets.  
: 
 We tackle the 3D grounding task in a challenging semi-supervised setting with limited annotations. We propose cross-task knowledge transfer to learn the models with labeled and unlabeled data. 
 
 Appendix:  


Decision: The paper introduces an innovative framework, named the 3D Cross-Task Teacher-Student Framework (3D-CTTSF), which integrates 3D visual grounding with dense captioning in a semi-supervised learning environment. This approach is novel for the 3D visual grounding task. The experimental results on established benchmarks validate the efficacy of the proposed framework. 

The paper receives the initial rating of two borderline accept and two weak accept. After the rebuttal, all reviewers are positive towards the paper, with the final scores of 5, 5, 5, 6, acknowledging the method’s novelty and effectiveness. The AC agrees with the rating and recommends accepting this paper.

Reviews: review: paper_summary: paper_summary: The main content of the paper is about a novel approach called "Cross-Task Knowledge Transfer for Semi-supervised Joint3D Grounding and Captioning". The paper proposes a framework that combines 3D visual grounding and dense captioning tasks in a semi-supervised learning setting. The framework utilizes a teacher-student architecture to transfer knowledge between the two tasks and generate pseudo-labels for unlabeled data. Experimental results demonstrate that the proposed framework achieves promising performance on various datasets.
strength_weakness: strength_weakness: It is novelty that bridges the gap between semi-supervised learning and fully-supervised methods for the joint tasks of 3D grounding and captioning. The proposed method is extensively evaluated on various datasets for both grounding and captioning tasks. The evaluation results demonstrate the effectiveness of the 3D-CTTSF in semi-supervised learning, showing improved performance compared to previous methods.
limitations: limitations: There is also another paper D3Net, which is also a paper to do 3D Dense Captioning and visual grounding. What are the differences between this paper and the D3Net? Perhaps, this paper is not the first work.
paper_summary: paper_summary: In this paper, authors propose a 3D cross-task teacher-student framework for joint 3D grounding and captioning in the semi-supervised setting. Experimental results on benchmarks prove the effectiveness of proposed framework.
strength_weakness: strength_weakness: 1.The written is clear and easy to follow.
2.The expansion to semi-supervised setting is reasonable and results seem promising.
3.high completion quality.
limitations: limitations: 1.The idea of leveraging cross-task knowledge transfer to enhance performance on the current task has been proven effective by previous works, even under semi-supervised learning settings.

2.The in-distribution setting based on ScanRefer's grounding annotations plus captioning annotations is clearly a straightforward and simple setup. Have the authors considered using a cross-domain 3D scene setup, aiding the in-distribution (ID) tasks with out-of-distribution (OOD) datasets, such as using outdoor scene point clouds and captioning to assist indoor scenes, or vice versa?

3.Given the in-distribution (ID) setting, the semi-supervised setting of this paper seems to have limited significance. If high-quality pseudo-labeling using cross-domain scene data could be achieved to further enhance performance, it would greatly improve the reliability of the method.

4.In the process of pseudo-labeling, could the label distribution of categories be taken into consideration? For instance, in ScanRefer, chairs are the most frequently occurring objects, while bathtubs are rare. After training with only 10% of the data, the model's generalization for bathtubs might be very limited. Therefore, the quality of pseudo-labels for these less frequently occurring objects could also be expected to be low on unlabeled data. Could you provide more insights?

5. Figure 2 is too complex and not easy to understand.
paper_summary: paper_summary: The paper proposed a semi-supervised training strategy for 3D visual grounding utilizing very few labels. The method relies on alternate 3D captioning and Grounding using a teacher-student training framework. Teacher warmed up to small scale labelled data;
Further Teacher produces pseudo object proposals and pseudo labels to train student.
strength_weakness: strength_weakness: The proposed method uses semi-supervised setting inside a teacher-student framework for learning 3D visual grounding. This setting is novel for 3D visual grounding task. The combination of captioning and grounding for 3D visual has been explored before as in [1]. In this work, it is extended to teacher-student training where the pseudo labels from teacher is used to train student on unlabelled point cloud data. The object and caption pseudo labels from teacher is carefully filtered to ensure quality of labels. Moreover data augmentation is differently applied to teacher and student to enable effective learning of student network.

The proposed work has some novelties compared to existing literature in 3D visual grounding
-	It addresses the problem of 3D visual grounding in a semi-supervised setting utilizing teacher pseudo labels to train student network
-	The design allows cross task knowledge transfer while learning  captioning and grounding   with pseudo labels .

The experimental results are performed satisfactorily with adequate comparisons to prove the usefulness of proposed method.  

[1] 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds, CVPR 2022.
limitations: limitations: A good pseudo label caption should ideally describe an object in the context of its  local neighborhood to differentiate it from similar objects in the scene. This characteristic is not coming out in the design. The ‘goodness’ of a pseudo label caption is decided by the ‘easiness’  to ground using the grounding module. However ,this can result in producing more descriptive captions of unique objects neglecting spatial layouts. It is not clear how this will be handled in the work.
paper_summary: paper_summary: The paper focuses on learning a 3D visual grounding model with minimal annotations by introducing a semi-supervised setting for the 3D visual grounding task. It proposes leveraging knowledge transfer from the related task of 3D captioning to improve the accuracy of 3D scene understanding. By training these tasks iteratively with unlabeled data, the authors aim to enhance object and text contexts within 3D scenes.
strength_weakness: strength_weakness: Innovative Approach: The paper introduces a novel semi-supervised setting for 3D visual grounding, addressing the challenge of limited annotated data.
Knowledge Transfer: Leveraging knowledge transfer from the 3D captioning task demonstrates a creative way to improve the performance of 3D scene understanding.
Iterative Training: The iterative training approach with unlabeled data shows promise in enhancing the object and text contexts within 3D scenes.
limitations: limitations: One specific limitation of the paper is the lack of information regarding the scalability and generalizability of the proposed 3D Cross-Task Teacher-Student Framework (3D-CTTSF). While the document discusses the effectiveness of the framework in a semi-supervised setting with limited annotations, it does not provide insights into how well the approach would perform across different datasets, varying levels of complexity in 3D scenes, or in real-world applications beyond the scope of the study. Understanding the framework's scalability and generalizability is crucial for assessing its potential impact and practical utility in diverse multimedia understanding tasks.