Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an interesting 'Hierarchical Part-Whole Attention' for multi-object tracking. The proposed module is integrated with transformer network and achieves good performance (comparable or even better results than SOTA mot trackers). The overall training efficiency is also good, i.e., 4 hours on 4*v100 GPUs, while other Transformer based trackers may need days. This paper is well-written and organized, and I believe it will be a good baseline for future works to compare and in-depth development on this framework. 
strength_and_weaknesses: strength: 
1. the idea of Hierarchical Part-Whole representation of target object in MOT seems interesting; 
2. the combination of Hierarchical Part-Whole Attention and Transformer works well on existing benchmark datasets; 
3. the paper is easy to follow and understand. 


weakness: 
1. the source code will be released or not is unclear. it is an interesting idea, but the implementation is also complicated. If the code is not available, maybe it is hard for other researchers to follow. 
2. the running efficiency is not real-time. i.e., not fast enough for practical applications. 
clarity,_quality,_novelty_and_reproducibility: this paper is clearly written and the information is enough to reproduce the experiments. 
summary_of_the_review: the idea is reasonable, 
the framework is not very complicated for MOT. 
the training time is acceptable, but the inference is fast; 
this paper is well-written and easy to follow.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a hierarchical representation of objects for multi-object tracking. The hierarchical representation consists of 3 levels: part, whole and union of overlapped objects. The proposed approach demonstrated good performance on multiple pedestrian/dance public datasets. 
strength_and_weaknesses: The strength of the paper is the fusing of the three levels of representation for multi-object tracking associations. The weaknesses of the paper are some of the notations and areas are not clearly presented.
1. The notations of M and N in section 3.1 are not defined when appearing in the first and second part. This causes confusion when reading the paper.
2. N should be T in 1<=i<=N in the third page.
3. In page 4, "we can tract the CNN features from them", what is the CNN architect to extract the features?
4. In page 6 Inference section, "whose negative value serves as the entries in the cost matrix". The cost value should not be some negative values. 

Another weakness of the approach is the lack of motion feature in the association. In the association, only appearance features are considered. This can lead to association errors when two objects have similar appearances (similar color of clothes) even they are farther apart in the space. In the Detection and Feature Extraction section, detection features are extracted from a video clip of T frames and then the detection from each one of the T frames is associated with the trajectories. There can be significant motions in the T frames and appearance would be hard to handle the discriminability of the objects.
clarity,_quality,_novelty_and_reproducibility: The combination of 3 levels of representation especially the union of overlap using attention is somewhat creative. 
summary_of_the_review: Overall, this work shows somewhat creativity, but only considers the appearance features can be insufficient to solve challenging tracking cases.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed modification for multiple object tracking framework based on GTR[Zhou et.al 2022]. Instead of use the feature representation from detection bounding boxes. This paper proposed to use split bounding boxes and surrounding image patch if there are other objects located close to the current object, which aims to tackle occlusion and avoid identity switch cases.

The proposed methods performs comparable with state-of-the-art methods on MOT17, MOT20 and DanckTrack benchmark. 
strength_and_weaknesses: Transformer based tracking framework demonstrated promising performance. Research effort to improve those framework should be encouraged. This paper tried to explicitly guide the model to learn local object patches and surrounding object patches to improve performance. Experiments demonstrated the proposed part-whole attention improves tracking performance, both in the ablation study and on MOT17 and DanceTrack comparing to GTR.

- How was the structure in part-whole attention module designed? Did the author try other designs? Any insights?
- Given this paper is based on GTR, experiments should conduct on TAO dataset.

clarity,_quality,_novelty_and_reproducibility: The overall structure is based on GTR but the context was not clear. The author should add add clear statement to acknowledge work by GTR paper. The proposed part-whole attention is well described. The discussion regarding L_feat should be expanded, especially alpha.

Experiments are conducted on different hardware setup and reported similar performance, which indicates certain level of reproducibility. But code is publicly available.

Modified a previous work to introduce more feature representation prior. Novelty is fair.
summary_of_the_review: This paper proposed Part-Whole Attention module for GTR framework for multiple object tracking task. The proposed module are clearly described and experiments demonstrated the effectiveness. 

More experiments on TAO dataset should be included to compare with its baseline method GTR. Discussions of the structure design for Part-Whole Attention module should be added
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** The paper proposes a hierarchical representation of objects for multi-object tracking. The hierarchical representation consists of 3 levels: part, whole and union of overlapped objects. The proposed approach demonstrated good performance on multiple pedestrian/dance public datasets.

**Strengths:** Promising results on the tested benchmarks.  The paper is easy to follow. 

**Weaknesses:** The following weaknesses were identified:

1. running speed is not real-time, thus not practical. [hWAz]
2. notations could be improved. [YxoG]
3. no motion features are used in association, which can cause errors when two objects have similar appearances [YxoG]
4. what is the design principle for the part-whole attention module? [rS4y]
5. Need experiments on TAO dataset to compare with GTR [rS4y]
6. more discussion on Lfeat [rS4y]

**Discussion:**
The authors wrote a response to address the concerns, and the reviewers maintained their positive ratings (668).
However, the AC noted that the technical/empirical ratings from Reviewers rS4y and YxoG: 
- Technical Novelty And Significance: 2: The contributions are only marginally significant or novel.
- Empirical Novelty And Significance: 2: The contributions are only marginally significant or novel.

After reading the paper, the AC agreed with the two reviewers that the novelty and technical contribution is quite low -- the main contribution is the particular way of performing cross-attention between object patches, whole object, and context to form a token representation of the object. However, the paper provides no analysis of the representation to show its suitability (e.g., visualizations of when it works and how it works), and the proposed representation is highly specific to MOT.  The rest of the MOT architecture using transformers follows the global association paradigm as in GTR (Zhou et al 2022) and VisTR (Wang et al 2021). The empirical results are also marginally significant -- the proposed method outperforms other transformer-based methods on 1 of 3 tested datasets (in terms of HOTA), although the proposed work is faster. However, there is not really an in-depth analysis of the speed-accuracy tradeoff to illustrate this. Given the similarity with GTR, the proposed method should have been compared on the TAO dataset (as suggested by rS4y), which GTR tests on. Finally, the proposed work includes the triplet loss (Eq 4), which is different from GTR, and there was no ablation study to show the effect of the extra loss term. Thus the real effect of the proposed representation is not well understood. 

Given the above, the AC found that the "6" ratings (marginally above acceptance threshold) from Reviewers rS4y and YxoG were inconsistent with the technical/empirical ratings.  When asked about this during the discussion, no reviewers offered to champion the paper. Therefore, the AC decided that "6" ratings were not appropriate, and thus decided to reject the paper. 

justification_for_why_not_higher_score: The technical novelty is very incremental, and very specific to MOT. The experiment results are mixed. The method is not fully tested against its closest baseline.  Some interesting ablation study is missing. 
justification_for_why_not_lower_score: n/a
Decision: Reject