Is Multiple Object Tracking a Matter of Specialization?
Authors: Gianluca Mancusi
Mattia Bernardi
Aniello Panariello
Angelo Porrello
Rita Cucchiara
Simone Calderara
Venue: NeurIPS 2024 poster (2024)
Keywords: Multiple Object Tracking
Parameter Efficient Fine-Tuning
Modular Deep Learning
Task Arithmetic
Primary area: deep_learning_architectures

Abstract: End-to-end transformer-based trackers have achieved remarkable performance on most human-related datasets. However, training these trackers in heterogeneous scenarios poses significant challenges, including negative interference - where the model learns conflicting scene-specific parameters - and limited domain generalization, which often necessitates expensive fine-tuning to adapt the models to new domains. In response to these challenges, we introduce Parameter-efficient Scenario-specific Tracking Architecture (PASTA), a novel framework that combines Parameter-Efficient Fine-Tuning (PEFT) and Modular Deep Learning (MDL). Specifically, we define key scenario attributes (e.g, camera-viewpoint, lighting condition) and train specialized PEFT modules for each attribute. These expert modules are combined in parameter space, enabling systematic generalization to new domains without increasing inference time. Extensive experiments on MOTSynth, along with zero-shot evaluations on MOT17 and PersonPath22 demonstrate that a neural tracker built from carefully selected modules surpasses its monolithic counterpart. We release models and code.

Is Multiple Object Tracking a Matter of Specialization?
Gianluca Mancusi
Mattia Bernardi
Aniello Panariello
Angelo Porrello
Rita Cucchiara
Simone Calderara
AlmageLab University of Modena and Reggio Emilia name. surname@unimon it
Abstract
End-to-end transformer-based trackers have achieved remarkable performance on most human-related datasets. However, training these trackers heterogeneous scenarios poses significant challenges, including negative interference where the model learns conflicting scene-specific parameters and limited domain gener- alization, which often necessitates expensive fine-tuning to adapt the models new domains. In response to these challenges, we introduce Parameter-efficient spenaii Tracking Architecture (PASTA), novel framework that com- bines Ceetnntte-Efffccienee Fine- Tuning (PEFT) and Modular Deep Learning (MDL). Specifically, we define key scenario attributes (e.g., vimeera-vewpooit, lighting condition) and train specialized PEFT modules for each attribute. These expert modules combined in parameter space, enabling systematic generalization new domains without increasing inference time. Extensive experiments on MOT- Synth, along with shot evaluations MOT17 and PersonPath22 demonstrate that neural tracker built from carefully selected modules surpasses its monolithic counterpart. We release models and code.
Introduction
Video Surveillance is essential for enhancing security, supporting law enforcement, improving safety, and increasing operational efficiency across various sectors. In this respect, Multiple Object Tracking (MOT) is widely studied topic due to its inherent complexity. Nowadays, MOT is commonly tackled with two main paradigms:
38 be-detcttonn methods have proven effective across multiple datasets, their performance struggles to scale on larger datasets due to the -in-differentiabee mechanism used for linking new detections to existing tracks. To this end, query-based methods are being employed to unify the detection and association phase.
(TbD) [1 28
43
29
or query-based
tracking [56] 58 63 [2] (i.e., swckingg-y--atteniion) Although
Nevertheless, training such end-to-end transformer-ban based methods presents significant challenges, as they tend to overfit specific scenario settings environments), require vast amounts of data 633 and incur substantial computational costs. Moreover, these methods degrade under domain shifts, struggling to outperform traditional TbD methods.
(e.g., camera viewpoint, indoor vs. outdoor
In light of these challenges, we propose novel framework, Parameter-efficien Scenario-specific Tracking Architecture (PASTA), aimed reducing the computational costs and enhancing the transfer capabilities of such models. Leveraging Parameter Efficient Fine- Tuning (PEFT) techniques [16] 34]
can significantly decrease computational expenses and training time, starting with frozen backbone pre-trane pre-trained on synthetic data. However, the model may still experience negative interference
49
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Modules Composition
Modules Deployment
End-to-End Tracker P. = 00
novel outdoor scene = Am
Domain Expert
Composed modules
Modules selection
Figure I: Given scene, select the modules corresponding to its attributes, such lighting and indoor/outdoor. These modules are composed and then deployed, yielding specialized model.
50 37 55]. phenomenon for which training multiple tasks (or scenarios) causes the model to learn task-specific parameters that may conflict. For instance, the model learns parameters tailored for indoor sports activity, it could detrimentally affect its performance depicting people walking. To this end, inspired by Modular Deep Learning (MDL) [5], we employ lightweight expert module for each attribute, learn them separately, and finally compose them efficiently [17]. This approach - depicted in Fig. -- is akin to chef preparing pasta dish. Each ingredient (i.e., module) prepared individually to preserve its unique flavor and then combined harmoniously to create a balanced dish. Moreover, as pasta must be perfectly al dente to serve as the ideal base for various sauces, the pre-trained backbone should be robust and well-tuned to serve as the foundation for the modules. These modules must be combined effectively to ensure the model performs well across diverse scenarios. Conversely, the result will sub-optimal incompatible modules are mixed - analogous combining ingredients that do not complement each other. Indeed, combining contrasting modules can lead ineffective handling of diverse tasks.
Notably, such a modular framework brings two advantages: it avoids negative interference and enhances generalization by leveraging domain-specific knowledge. Firstly, starting from a pre-trained backbone, we train each module independently to prevent parameter conflicts, ensuring that gradient updates confined the relevant module for the specific scenario. This assures that parameters learned for one attribute do not negatively impact the performance of another. Secondly, the modular approach allows us to exploit domain knowledge fully, even when encountering novel attribute combination. Indeed, as shown in Sec. 5.3 our approach is effective even in zero-shot setting (i.e., without further fine-tuning on the target dataset). Moreover, the selection of the modules may be done automatically or in more realistic production environment by video surveillance operators.
To evaluate approach, conduct extensive experiments the synthetic MOTSynth (10] and the real-world MOTI7 [ and PersonPath22 [44) datasets. The results show that PASTA can effectively leverage the knowledge learned by the modules improve tracking performance both the source dataset and in zero-shot scenarios. To summarize, we highlight the following main contributions:
We propose PASTA, a novel framework for Multiple Object Tracking built on Modular Deep Learning, enabling the fine-tuning of query-based trackers with PEFT techniques.
By incorporating expert modules, we improve domain transfer and prevent negative interfer- ence while fine-tuning MOT models.
Comprehensive evaluation confirms the validity of our approach and its effectiveness in zero-shot tracking scenarios.
2
Related works
Multiple Object Tracking, The most widely adopted paradigm for Multiple Object Tracking (MOT) tracking- benkking-by-deeerrtee by-detection (TbD) 51 61,
38). First, an object detector (e.g., YOLOX (14) localizes objects in the current frame. Next, the association step matches detections to tracks from the previous frame by solving a minimum-cost bipartite matching problem, with the association cost defined in various forms (e.g., loU 3 61 GloU [40, or geometrical cues 29 33 ). This pairing
28
43
typically occurs immediately after propagating the previous tracks the current frame using a motion model (e.g., Kalman Filter [19). Notably, methods following such paradigm have succeeded on

---

complex human-related MOT benchmarks 9 47 44]. In TbD, the detection and data-association steps are equally crucial accurately localizing and tracking objects. Recent works [65| have attempted to unify these steps; however, progress toward fully unified algorithm was constrained by
a significant limitation - the data association process (e.g., the Hungarian algorithm (0) is inherently
non-differentiable. An initial effort was made by Xu et al. (53) that proposed a differentiable version of
the Hungarian algorithm, later advanced by end-to-end transformer-basee trackers [31 58 63 [56 [3]
However, transformer-base trackers (also known as backing-by-atteniin data to achieve decent generalization capabilittes [58 BT] Due to the data scarcity in MOT, these models often overfit to the specific domain they were trained on, which hampers their ability to generalize to different domains [17 37).
require large amounts of
Modular Deep Learning (MDL.. Considering recent trends in the field of deep learning, state- of-the-art models have become increasingly larger. Consequentially, fine-tuning these models has become expensive; concurrently, they still struggle with tasks like symbolic reasoning and temporal understanding [35]. Recent learning paradigms based on Modular Deep Learning (MDL) [35] can address these challenges by disentangling core pre-training knowledge from domain-specific capabilities. By applying modularity principles, deep models can be easily edited, allowing for the
seamless integration of new capabilities and the selective removal of existing ones [26 36).
Specifically, lightweight computation functions named modules are employed to adapt a pre- trained neural network. To do so, several fine-tunin techniques could be used to realize these modules, such LoRA [16), (IA) (25). and SSF 23 These multiple modules can be learned on different tasks such that they can specialize in different concepts (32. At inference time, not all modules have to be active at the same time. Instead, they can be selectively utilized as needed, either based on prior knowledge of the domain or dynamically in response to the current input. To establish which modules activate, it common practice rely on routing function, which can be either learned or fixed. Finally, the outputs of the selected modules are combined using an aggregation cocuutioonunnttoonn function. To minimize inference costs, this process is usually performed in the parameter space rather than the output space, an activity often referred to as model merging [54]. Specifically, single forward pass is performed using weights generated by a linear combination of those selected by the routing function.
Domain adaptation and open-vocay approaches in MOT. Currently, domain adaptation tech- secking-by-detection methods, with GHOST (43] and DARTH [42]
niques have only been applied serving as notable examples. In particular, GHOST adapts the visual encoder employed to feed the appearance model by updating the sufficient statistics of the Batch Normalization layers during inference. In contrast, our approach regards rrackin-by--ttteen by-attention approaches and adapts the entire network. Moreover, DARTH employs test-time adaptation (TTA) [24] and Knowledge Distillation, requiring multiple forward passes and entire sequences, making it computationally heavy and less practical for real-time use. In contrast, our method is entirely online and requires only basic target scene attributes, with no further training during deployment.
Recent advances in zero-shot tracking have focused open-vocabulary tracking, where the model can track novel object categories by prompting it with the corresponding textual representation. In this respect, methods like OVTrack (22) and Z-GMOT (48) leverage CLIP [39] and language- based pre-training, while OVTracktor 7 extends tracking to any category. Our method does not use open-vocabulary models but emphasizes domain knowledge transfer in end-to-end trackers.
3 Preliminaries
Efficient fine-tuning. Given the substantial size of recent vision backbones, often consisting of hundreds of millions of parameters, adapting them to new scenarios is computationally expensive, both in terms of time and memory requirements. To tackle the above problems, Parameter Efficient Fine-Tuning (PEFT) started to take place in recent literature. Among these methods, Low- Rank Adaptation (LoRA) (16) excels at such purpose. Specifically, LoRA adapts pre-trained weight matrix On E Ra with and & being the dimensions of the matrix, by leveraging a low-rank decomposition 0, + A0 = On + BA, where B E Rdxr, A I and r & min(d, k). During training, O is kept frozen, while the smaller A and B matrices are instead trainable, making the process highly efficient. The forward pass becomes h Our + BAr, where are the input features.
3

---

Scenario
Parameter Modules Library
Decoder
Good / Rad Lighting
Layar
Lagan
High Viewpoint
Encoder
Domain Expert
Selft Jesent OP
High Low Occupancy
Deformazle
In/ Outdoor Location
Feed Forwwnt
Selected
Deployment
PEFT applied
Moving/ Static Camera
Λ
Modules Composition
Track querles
Detect queries
Figure 2: Overview of our modular architecture. A domain expert selects PEFT modules based sequence attributes such lighting and camera movement. These selected modules then composed and applied to each model layer, adapting the backhone and encoder-decoder architecture.
Query-based Multiple Object Tracking. The underlying backbone of our transformer based tracker follows the structure of [58). In a nutshell, such a query-based model forces each query to recall the same instance across different frames. Specifically, we leverage end-to-end trainable tracker built upon the Deformable DETR (6) framework conditioned by the image features extracted with a convolutional backbone (i.e., ResNet [5).. Following [63] further condition the DETR decoder with set of detections from an external detector network and a shared learnable query.
At time O, new proposals are generated from the objects detected in the scene. These proposals are then updated through self-attention and interact with image features via the deformable attention layer. The final prediction output is the summation of the initial anchors and the predicted offsets. For subsequent frames (f > 0), track queries generated from the previous frame are concatenated with learnable proposal queries of the current frame. Moreover, previous predictions with current proposals to establish new anchors for the incoming frame. We refer the reader to the original paper [63] for further details. It is noted that the flexibility of this architecture allows for the seamless integration of techniques based on modularity.
integrated
4
Method
We herein present PASTA, depicted in Fig. 2] novel approach Multiple Object Tracking that leverages PEFT modules to enable sttribute-speciff module specialization and reuse. This approach allows for the dynamic configuration of an end-to-end tracker by selecting the appropriate modules for each input scene, fully leveraging heterogeneous pre-training while avoiding negative transfer.
Attribute- based modularity. We devise a set of learnable modules to fine- fine-tune each layer of our query-based tracker. Each module is related to attribute: shown in Fig. 3 we define N attributes, namely lighting, viewpoint, occupancy, location, and camera motion, and provide a tailored module for each discrete value these attributes take (see Sec. [.3 for details). For instance, the location attribute has indoor and outdoor modules. At inference time, prior knowledge about the input scene used to determine the appropriate value for each attribute, which in turn selects the corresponding modules from the "inventory", denoted as M.
Since the base model 663 relies on heterogeneous layers namely, convolutional (e.g., ResNet) and attention-basee blocks (e.g., Deformable DETR) we employ two different strategies fine-tune the modules. Specifically, after each convolutional layer of the ResNet backbone, we apply strategy

---

Indoor
Low
Moving
Low
Good
Outdoor
High Angle
Crowded
Good
Outdoor
Maving
Low
Angle
Density Lighting
Camera
Lighting
Angle
Density Lighting
Figure 3: Examples of surveillance scenes and their corresponding attributes used by PASTA.
that learns channel-wise scale and shift parameters; for each layer of Deformable DETR, instead, we employ LoRA- -based fine-tuning each linear layer. In formal terms, considering each convolutional layer of the ResNet backbone, we deploy [M pairs (2/m Sm |Md|| of learnable vectors y, 3 RC, where C' is the number of the output channels. For each linear layer of the encoder-decoder structure underlying Deformable DETR, we devise |M| pairs {Am, Bm MM of learnable LoRA matrices.
During training, we start with the pre-trained weights and integrate all the modules while keeping the original parameters frozen. To prevent negative interference, we optimize each module independently, randomly sampling one attribute at time and updating only the corresponding module at each training iteration. By the end of the training process, we obtain a set of specialized parameters (experts), which can be seamlessly merged during inference to improve overall tracking performance.
Routing through Domain Expert. During inference, two essential steps are required to exploit the learned modules: routing and aggregation. With multiple modules available from the inventory M, a routing strategy is required determine the modules that should be active. To make this selection, we draw on what is known in the literature as expert knowledge [52, 35 (or "Domain Expert" in Fig. 2 In real-world applications such as video analyties, the expertise guiding the selection can come from a video surveillance operator or human analyst, who configures the appropriate modules to reflect domain- and scene-specific settings, such as camera perspective, lighting conditions, and other critical details. This approach allows users to optimize the tracking module for their unique contexts without extensive retraining. Additionally, the modular nature of the system enables easy integration of new modules to address emerging attributes or scenarios.
Relying on Domain Expert to select attributes is grounded practice in real-world applications. For instance, the camera's mounting perspective and whether the scene is indoors or outdoors are typically known factors in fixed-camera scenarios. Additionally, automatic approaches can be envisioned to minimize human intervention further. For example, lighting conditions can be inferred by analyzing brightness levels, and detector can count objects of interest in the scene, classifying crowd density.
Modules composition. In the final step, we aggregate the selected modules ("Modules Composition" in Fig. ( and incorporate the result into the pre-trained tracker to create an expert model. Since these modules have been obtained by fine-tuning from On each module 0- corresponds to a specific displacement r* 0 O, in parameter space relative to the initial pre-training parameters 0g This displacement is known as the task vector [18). The final composed model f(; 0.) is defined as:
f(::0.) where O. = Ou IANAA
Σ. and 7, E M.
(1)
When A, = N. the formula simplifies to the average of the task vectors corresponding to each attribute. We employ this straightforward strategy for X; giving equal weight to all attributes. However, considering the task vector т; associated with the i-th attribute, we employ a more sophisticated approach. If there same dataset, such as MOTSynth), the task vector T; is simply set to the displacement 7* produced by the expert module selected by the Domain Expert. In contrast, when domain shifts are present (e.g., training on MOTSynth and testing on MOT17), we adopt a soft strategy that considers all the modules in the inventory associated with the relevant attribute. In doing so, follow the insights from [59]. where the authors demonstrated that scenarios with shifting tasks benefit from richer representations than those derived from a single optimization episode.
domain shifts during inference (i.e.. both training and testing occur on the
Specifically, given the i-th attribute, R(i) be the set of its modules. We recall that each attribute admits multiple discrete values (e.g., R(occupancy) = ["low", "medium", "high")), and different
5

---

attributes may have different cardinalities (e.g.. | [ 3 and [[[lighinng]= 2, as detailed in Sec. (5.3). Building on this, we employ soft routing to create the corresponding task vector, assigning the largest portion of the cake, e.g- p 0.80, to the module selected by the Domain Expert. ensuring that the total sum equals 1.
The remaining modules are weighted by (1 - For example, considering those layers fine-tuned with the LoRA, the corresponding task vector is computed as:
if m is selected, otherwise.
= EmER()) Am Am,
where
Am
(2)
Note that when o 1, the soft strategy becomes hard, meaning that only the module selected by Domain Expert is utilized. By applying the formula above to all attributes, we obtain N task vectors, which we aggregate following Eq. (
Similarly, we apply channel-wise scale and shift 23 operations to adapt each backbone layer. Formally, given the output F of a convolutional layer, the i-th module applies scale & shift operation to obtain the edited F;, such that F; = @ F + 8 with @ denoting the Hadamard product. At inference time, we combine the output of different scale shift modules by noting that
i ²= - (1 & F + :.) = Ei >4 o F) + X,3; EEi" Ar71)) ④ F E A,3, (3) which means that parametrizing the scale & shift layer with a simple weighted average effectively results in averaging the outputs of the corresponding individual layers. The formula above applies to the in-domain setting but can be easily generalized to the soft routing scheme outlined by Eq. (2) Eventually, as discussed [23] the scale & shift layer can be absorbed into the previous projection layer, thus ensuring that the inference process incurs no additional computational costs. The same
re- re-parametrization trick can be employed to extract the task vector underlying scale & shift fine-tuning (refer to appendix A for additional notes).
Experiments
5.1 Datasets
MOTSynth 10] a large synthetic dataset for pedestrian detection and tracking in urban scenarios, generated using photorealistic video game. It comprises 764 full HD videos, each 1800 frames long, showcasing various attributes. our experiments, following [29], we reduced the test sequences to 600 frames each and further split the training set to extract 48 validation sequences, shortened to 150 frames, for validation during training.
PersonPath22
is a large-scale pedestrian dataset consisting of 236 real-world videos featuring longer occlusions and more crowded scenes. It is divided into 138 training videos and 98 test videos.
MOT17 g is a well-known benchmark, containing sequences for training and for testing, with different image resolutions, featuring crowded street scenarios with both static and moving cameras.
5.2 Experimental setting
We evaluate our proposed PASTA both in-domain and out-of-domain scenarios. For the in- domain evaluation, we train and test PASTA on the MOTSynth synthetic dataset (Sec. (.4) using expert modules domain-specifie context. As baseline, train (63) on MOTSynth without using modules, referring to this model as MOTRv2-MS. For the out-of-domain evaluation, we conduct a synth-to-real sero-shot experiment on MOT17 and PersonPath22 (Sec. 5.5). Starting from training on MOTSynth, we test PASTA on these datasets without additional training, showcasing its ability to generalize under non-identically distributed domains. Finally, we present series of ablation studies in Sec. O to take closer look at the effectiveness of our method.
Competing trackers and metrics. In addition, we report the performance of other notable methods, including strong rrakkig-by--deteee by-detection baselines such as ByteTrack [6]] and OC-Sort [5]. We also
include evaluations of query-based trackers, such as TrackFormer [3 and MOTRv2 [33 (see
MOTRx2-MS), To compare their performance, we employ five metrics, ordered from detection to association, as recommended by 42 These metrics are DetA 27 MOTA (2) HOTA (27). IDF1 41, and AssA 277. For the PersonPath22 dataset, we use their official metries, MOTA and IDF1, supplemented by FP (false positives), FN (false negatives), and IDSW (identity switches).

---

Table Evaluation on MOTSSynt test set. |
the number of trainable parameters.
O
HOTA
IDFIt
MOTA
DetAf
AssAt
SORT
3
46.0
55.7
50.9
49.9
42.8
Byte'Track 61
45.7
56.4
61.8
50.1
41.9
OCSort S
46.9
56.8
59.1
48.7
45.6
TrackFormer
44M
41.3
49.9
47.7
44.4
40.6
MOTR<2-MS
42M
52.4
56.6
61.9
56.4
49.0
PASTA (Ours)
15M
53.0
57.6
62.0
56.2
50.4
5.3 Implementation details
We initialize our models using the pre-trained weights from DanceTrack [47], provided by the authors of [63]. We employ YOLOX (14) as the auxiliary detector, exploiting weights from ByteTrack [61] To provide shared initialization for both PASTA and MOTRv2-MS training, train a bootstrap model starting from the Dance'Track pre-train for 28k iterations on the MOTSSyth training set. This bootstrap initialization uses half of the original training sequences from MOTSynth to align our model with the scenarios represented in the dataset. The learning rates are set to x 10 for the transformer and 1 10 - for the visual backbone.
In the second phase, we deploy the PEFT modules to fine-tune the bootstrap model. By excluding half the sequences during the bootstrap, we make sure that the modules still learn valuable features. To ensure fair comparison, we train each module for similar number of iterations as MOTRv2-MS, with approximately 17k iterations. Regarding the encoder-decoder model, we apply our modularization strategy to every linear layer except those with output dimension less than 128. For the LoRA hyperparameters, we The scale & shift layers employ a learning rate of 1 x 10 and weight decay of 1 x 10 -4 The training performed on single RTX 4080 GPU with batch size of for both phases. Due the small batch size, we accumulate gradients over four backward steps before performing an optimizer step. Each module is trained independently on the entire MOTSynth training set. With 12 modules, our model has approximately 15 million trainable parameters.
16, weight decay of 1. and learning rate of 10-1
Attributes. We employ five key attributes to realize our modular architecture: lighting, camera viewpoint, people occupancy, location, and camera motion. For lighting, specialize modules for good and bad lighting conditions. To do so, we threshold the brightness value V of the HSV representation at 70. The viewpoint attribute includes modules for high, medium, and low camera angles. We manually annotate this attribute as follows: i) scenes where the camera is parallel to the ground or below pedestrian head level are labeled as "low-level"; üi) "high-level" viewpoints include vertical perspectives or scenes where the camera is positioned very high or far from people; and iii) "medium-level" includes all other camera angles. For occupancy, design modules that reflect the crowd density within the scene: low (up to 10 people), medium (10 to 40 people), and high (more than 40 people), based the count of detections with confidence score above 0.2. The location attribute differentiates between indoor and ouidoor settings. Lastly, the motion attribute comprises modules for both moving and static cameras, enabling the model adapt to different camera movement scenarios. Further details on dataset statistics are provided in appendix C
5.4 Performance the in-domain setting
To assess the impact of the negative interference, we conduct several experiments on MOTSynth (see Tab. [T. Given the wide variety of scenarios in such synthetic dataset, one can appreciate the advantages of using specialized modules. Indeed, integrating our modules resulted in an overall im- provement w.r.t. its fine-tuning counterpart (MOTRv2-MS3 Specifically, we observe an improvement over the association metrics (AssA, IDF1) and the HOTA and MOTA metrics. These enhancements suggest the benefits of our approach in reducing negative interference during training. By assigning each module a specific role tailored to particular scenario settings, we achieve improved training stability through deterministic selection process guided by domain expert.
7

---

Table 2: Zero-shot evaluation on MOT17. PASTA is evaluated in zero-shot by selecting the best attributes on the source dataset.
HOTA!
IDFIt
MOTAT
DetAf
AssA
fully-trained
SORT 3
64.3
73.1
70.9
63.3
66.1
OC-SORT
66.4
77.8
74.5
64.1
69.1
TrackFormer
74.4
71.3
BrtcTrack 61
67.9
79.3
76.6
66.6
69.7
MOTRv2 63
66.8
78.9
73.2
62.5
71.4
zero-shot
TrackFormer 31
51.0
63.9
58.7
51.8
61.2
MOTRv2-MS
62.6
73.0
67.6
60.3
65.5
PASTA (p = 1)
63.7
74.1
67.9
60.3
67.9
PASTA (p = 0.8)
64.0
74.9
68.1
60.4
68.3
Table 3: Evaluation on PersonPath22 test set. PASTA is evaluated in zero-shot by selecting the best attributes on the source dataset.
MOTAT
IDF1
FP4
FN4
IDSW
fully-trained
CenterTrack 65
59.3
46.4
240300
555
10319
SiamMOT 45
67.5
53.7
13217
62.543
8942
FairMOT 662
61.8
61.1
14550
80034
5095
IDFree 46
68.6
63.1
9218
663533
6148
TrackFormer
69.7
57.1
23 138
303
8633
ByteTraack [61]
75.4
66.8
17214
90
5931
zero-shot
TrackFormer 31
39.2
43.3
21 402
126 082
10023
MOTRy2-MS
48.3
53.1
28 483
98 007
7154
PASTA (p- 1)
49.7
53.7
18211
105 611
6321
PASTA (p 0.8)
50.0
53.8
18 038
105 454
6037
5.5 Performance the out-of-domain setting
By designing distinct modules for various input conditions, we can effectively select the appropriate modules to handle distribution shifts, such as transitions to new domain. We assess the benefits of this ability using synthetic data for training, and then evaluate on new, unseen datasets without any additional re- re-training (zero-shot). To do this, we start with our model trained on MOTSynth as described in Sec. 5.4 and evaluate it on MOT17 (Tab. 2 and PersonPath22 (Tab. B) While these datasets share similarities in the attributes we employed, we emphasize that the source dataset is synthetic and the targets are real-world, resulting significant shift.
The results reported in Tab. 2 and show an improvement over the baseline (i.e., MOTRx2-MS), with +1.4 in HOTA and +1.9 IDFI in zero-shot MOT17, and +1.7 MOTA and +0.7 IDFI in PersonPath22 Our approach demonstrates better generalization capabilities, helping close the gap with fully-trained methods while less computationally demanding. These results indicate that modularity enhances performance within the source dataset and improves domain generalization, leading to more reliable and versatile approach for tracking. Furthermore, other than reporting the results with the standard module selection (considering only the modules present in the scenes, 1). we also experiment with the weighted aggregation of all modules (p 0.8) (detailed Sec. 4 Interestingly, while the standard strategy shows improvements, the weighted aggregation strategy yields better performance. This suggests that richer representations, obtained by including multiple modules per attribute, more effective for sero-shot scenarios than single-mouule approach [59].

---

Table Zero-shot evaluation of PASTA trained on MOT17 and tested on PersonPath222. PASTA is evaluated in zero-shot by selecting the best attributes on the source dataset.
MOTA1
IDF11
FP4
FN1
IDSW
Sully-trained
TrackFormer 31
69.7
57.1
23 138
47 303
8633
Brte'Track [6]
75.4
66.8
17 214
40902
5931
zero-shot
MOTRv2-MS
43.9
51.5
8304
119 391
5342
PASTA
46.1
54.6
7895
114 620
4702
Table Ablation study on different module aggregation and selection strategies. (Left) MOTSynth validation, (Right) Zero-shot on MOT17 validation. The strategy select is highlighted in yellow.
MOTSynth (val)
HOTAI
IDFIt
MOTA
MOT17 (val)
HOTAI
IDFIt
MOTA!
aggregation
aggregation
Sum (only selected)
0.65
0.44
-0.69
Sum (only selected)
0.58
0.41
-0.03
Weighted avg. (p 0.8)
59,9
66.8
59.6
Weighted avg. (p 0.8)
64.0
74.9
68.1
Mean avg. 1p 1.0)
60.1
67.2
59.9
Mcan avg. (p 1.0)
63.7
74.1
67.9
selection
selection
Opposite modules
59.2
66.5
58.7
Opposite modules
62.9
73.9
67.1
All modules Domain Expert
59.8
67.0
59.4
All modules
63.1
74.1
67.7
60.1
67.2
59.9
Domain Expert
63.7
74.1
67.9
Evaluating zero-shot real-to-real transfer. In Tab. we present an additional experiment to evaluate the performance of PASTA in zero-shot setting, this time using realistic dataset as the source, rather than a synthetic one. For comparison, we train MOTRv2 on the MOT17 dataset and assess its performance on PersonPath22. Our approach showcases superior results compared to the fine-tuned MOTRv2, highlighting that leveraging modules enhances the model, with improved generalization capabilities in new and real-world domains.
6 Ablation studies
In Tab. 5] we evaluate the effect of various routing and aggregation strategies in both the in-domain setting (MOTSynth, left side of Tab. 5) and the zero-shot setting (MOT17, right side of Tab. (5). In the in-domain scenario, the results show that averaging the modules selected by the Domain Expert, specifically using Mean avg. (p- = 1.0), is the most effective strategy. We also experimented with summation, proposed by [60]. but this method produced bad results, which we impute to the alteration of weight magnitudes when summing multiple modules. Another noteworthy approach is the weighted avg., described in Sec. which incorporates all modules, including those not selected.
While using only the selected modules is the optimal strategy in the in-domain scenario, for the zero-shot case (MOT17), incorporating knowledge from the non-selected modules specifically, using Weighted avg. (p = 0.8) enhances tracking performance. This pattern also consistent when the domain shift involves evaluation on the PersonPath22 dataset (see appendix E
Module selection. Should we select only the modules representing the current scenario, as deter- mined by the Domain Expert approach, or would performance improve by incorporating all available modules? In Tab. 5 we investigate this matter by comparing these two approaches. To provide more comprehensive perspective, we also evaluate strategy that, in stark contrast to the Domain Expert, selects the opposite modules (e.g., selecting the outdoor and poor lighting modules when presented with an indoor, well-lit scene). The lowest performance is observed when using opposite modules, indicating that using the proper modules provides valuable information about the current scene. Interestingly, the model still performs relatively well despite using opposite attributes, likely due to contributions from other modules whose general knowledge the domain sustains overall performance. This suggests that modules can assist one another in solving tasks. Moreover, reduced

---

57.41
57.45
57.48
57.81
Table Opposite modules selection.
56
56,87
HOTAЛ
IDF1t
MOTA
54
54.97
No modules
58,2
64.9
55.6
52.48
52.79
52.74
52.86
Opposite: only lighting
58.9
66.2
57.8
52
50.63
Opposite: only viewpoint
58.8
65.7
57.0
HOTA
Opposite: only occupancy
58.6
66.4
59.2
base model lighting
occupancy mation
location
viewpoint
Opposite: only location
66.0
58.4
+
Opposite: only camera
58.5
58.4
Average opposite
58.7
66.0
58.2
Figure 4: IDFI and MOTA when adding new attributes MOTSSnth.
Correct modules
60.1
67.2
59.9
Table Performance comparison of our approach without applying fine-tuning and to specific parts of the architecture (i.e., decoder, encoder, visual backbone).
Fine-tuning applies
HOTA
IDF1
MOTA
DetAf
AssA
none
52.4
56.6
61.9
56.4
49.0
all except the decoder all except the encoder all except the backbone PASTA (all)
51.5
56.0
58.9
53.6
49.8
52.4
56.9
61.2
55.7
49.7
52.5
57.0
61.5
55.6
49.9
53.0
57.6
62.0
56.2
50.4
negative interference achieved by training each module separately prevents the modules from relying on each other and allows them make unique contributions independently.
Furthermore, Fig. we illustrate how the incremental addition of specialized modules improves IDF1 and HOTA metrics, showcasing that greater specialization of the modules gradually enhances overall performance. For a more detailed analysis, in Tab, 6 we select the opposite modules instead of the correct one for each attribute. Although the metrics are further reduced, the model performs well due to its robust pre-tranning as indicated by the no modules baseline shown in the table.
Block-wist analysis. In our approach, attribute-related modules are applied to edit the entire network. However, users may opt edit selectively specific parts of the architecture, thereby identifying which components are most critical. In Tab. 7 we conduct an ablation study by excluding our modules from being applied to varying components of the architecture. The results indicate that applying task vectors to the decoder significantly degrades detection and association metrics. We believe that this degradation can be explained by considering the crucial role of the decoder. The decoder must indeed gather information from detection. tracking, and proposal queries while simultancously integrating visual information from the encoder. Consequently, not adapting the decoder prevents the architecture from effectively leveraging queries and visual cues. The encoder also contributes substantially, though to lesser extent than the decoder, as primarily refines and contextualizes visual features from the backbone. Finally, the backbone shows the smallest contribution.
7 Conclusions
In this work, we introduce PASTA, a novel framework that enhances domain generalization in tracking-by-query methods for Multiple Object Tracking. Our approach features a modular struc- ture with dedicated modules tailored different attributes of real-world scenes. These modules utilize Parameter- Efficient Fine- TaneTTnnig techniques, enabling the integration of scene-specific parame- ters while minimizing computational load. Comprehensive experiments demonstrate that domain- specialized modules significantly bolster robustness, allowing effective adaptation across domains without extensive retraining. PASTA further enables camera operators to configure the optimal module for each unique scenario, ensuring precise adaptation to diverse real-world settings.
10

---

Acknowledgements
The research activities conducted by Angelo Porrello were funded by the Italian Ministry for Univer- sity and Research under the PNRR project ECOSISTER ECS 00000033 CUP E93C22001 100001. Additionally, the research carried out by Rita Cucchiara was supported by the EU Horizon project "ELIAS European Lighthouse of Al for Sustainability" (No. 101 100377..
References
[1] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In IEEE International Conference on Computer Vision, 2019.
[2] Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing, 2008
[3] Alex Bewley, Zongyuan Ge. Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In IEEE International Conference on Image Processing, 2016.
[4] Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard. Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. In Transactions on Mackine Learning Research, 2024.
[5] Jinkun Cao, Xinshuo Weng, Rawal Khirodkar, Jiangmiao Pang, and Kris Kitani. Observation-centric sort: Rethinking sort for robust multi-object tracking. In Proceedings of the IEEE conference on Computer Vision and Pottern Recognition, 2022.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229 Springer, 2020.
[7] Wen-Hsuan Chu, Adam W. Harley, Pavel Tokmakov, Achal Dave, Leonidas J. Guibas, and Katerina Fragkiadaki. Zero-shot open-vocabulary tracking with large pre-trained models. In arXiV preprint arXiV:2310.06692, 2023.
[8] Patrick Dendorfer, Aljosa Osep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leii- Motchallenge: A benchmark for single-camera multiple target tracking. International Journal of Computer Vision, 2021.
[9] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé, Mot20: A benchmark for multi object tracking crowded seenes. arXiv preprint arXiv:2003.0 09003, 2020.
[10] Matteo Fabbri, Guillem Brasó, Gianluca Maugeri, Aljoša Ošep, Riccardo Gasparini, Orcun Cetintas, Simone Calderara, Laura Leal-Taixé, and Rita Cucchiara. Motsynth: How can synthetic data help pedestrian detection and tracking? In IEEE International Conference Computer Vision, 2021.
[11] Emanuele Frascaroli, Aniello Panariello, Pietro Buzzega, Lorenzo Bonicelli, Angelo Porrello, and Simone Calderara. Clip with generative latent replay: strong baseline for incremental learning. In British Machine Vision Conference, 2024.
[12] Ruopeng Gao and Limin Wang. Memotr: Long-term memory-augmenter transformer for multi-object tracking. In IEEE International Conference Computer Vision, 2023.
[13] Ruopeng Gao, Yijun Zhang, and Limin Wang. Multiple object tracking as id prediction. arXiv preprint arXiv:2403. 16848, 2024,
[14] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv: 200.:2100.000 100.08430, 2021.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016.
[16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations Workshop, 2021.
[17] Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni Compositionality decomposed: how do neural networks generalise? Journal of Artificial Intelligence Research, 2019.
11

---

[18] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman. Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. International Conference Learning Representations Workshop. 2022.
[19] Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. ASME Journal of Basic Engineering. 1960.
[20] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
[21] B. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence- io-sequence recurrent networks. International Conference on Machine Learning, 2017.
[22] Siyuan Li, Tobias Fischer, Ke, Henghui Ding, Martin Danelljan, and Fisher Yu. Ovtrack: Open- vocabulary multiple object tracking. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2023.
[23] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109-123, 2022.
[24] Jian Liang, Ran He, and Tieniu Tan. comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, pages 1-34, 2024.
[25] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin Raffel. Few-shot parameter-effiiient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 2022.
[26] Tian Yu Liu and Stefano Soatto. Tangent model composition for ensembling and continual fine-tuning. In IEEE International Conference on Computer Vision, 2023.
[27] Jonathon Luiten. Aljosa Osep. Patrick Dendorfer, Philip Torr, Andreas Geiger. Laura Leal- Taixé, and Bastian Leibe. Hota: A higher order metric evaluating multi-object tracking, International Journal of Computer Vision, 2021.
[28] Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang. Rethinking pseudo-lidar representation. Proceedings of the European Conference on Computer Vision, 2020.
[29] Gianluca Mancusi, Aniello Panariello, Angelo Porrello, Matteo Fabbri, Simone Calderara, and Rita Cucchiara. Trackflow: Multi-object tracking with normalizing flows. In IEEE International Conference on Computer Vision, 2023.
[30] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, 1989.
[31] Tim Meinhardt, Alexander Kirillov, Laura Leal- Taixe, and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2022.
[32] Guillermo Ortiz-Jimenes Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
[33] Aniello Panariello, Gianluca Mancusi, Fedy Haj Ali, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Distformer: Enhancing local and global features for monocular per-object distance estimation.
arXiv preprint vrXiv:2401.03199 2024.
[34] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville, Film; Visual reason- ing with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[35] Jonas Pfeiffor, Scbastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. Modular deep learning. In Transactions on Machine Learning Research, 2023.
[36] Angelo Porrello, Lorenzo Bonicelli, Pietro Buzzega, Monica Millunzi. Simone Calderara, and Rita Cucchiara, A second-order perspective on model compositionality and incremental learning. arXiv preprint arXiv:2405. 16350, 2024.
[37] Zheng Qin, Le Wang, Sanping Zhou, Panpan Fu, Gang Hua, and Wei Tang. Towards generalizable multi-object tracking. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition. 2024.
12

---

[38] Zheng Qin, Sanping Zhou, Le Wang, Jinghai Duan, Cang Hua, and Wei Tang. Motiontrack: Learning robust short-term and long-term motions for multi-object tracking. In Proceedings of the IEEE conference on Computer Vixion and Puttern Recognition, 2023.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[40] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2019.
[41] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi-target, multi-camera tracking. In Proceedings of the European Conference on Computer Vision, 2016.
[42] Mattia Segu, Bernt Schiele, and Fisher Yu. Darth: Holistic test-time adaptation for multiple object tracking. In IEEE International Conference on Computer Vision, 2023.
[43] Jenny Seidenschwarz, Guillem Brasó, Victor Castro Serrano, Ismail Elezi, and Laura Laix- Simple cues lead to a strong multi-abject tracker. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2023.
[44] Bing Shuai, Alessandro Bergamo, Uta Buechler, Andrew Berneshawi, Alyssa Boden, and Joseph Tighe. Large scale real-world multi-person tracking. In Proceedings of the European Conference Computer Vision, 2022.
[45] Bing Shuai, Andrew Berneshawi, Xinyu Li. Davide Modolo, and Joseph Tighe. Siammot: Siamese multi-object tracking. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, 2021.
[46] Bing Shuai, Xinyu Li, Kaustav Kundu, and Joseph Tighe. Id-free person similarity learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2022.
[47] Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. Dancetrack: Multi- object tracking in uniform appearance and diverse motion. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2022.
[48] Kim Tran, Anh Duy Le Dinh, Tien-Phat Nguyen, Thinh Phan, Pha Nguyen, Khoa Luu, Donald Adjeroh, Gianfranco Doretto, and Ngan Le. Z-GMOT: Zero-shot generic multiple object tracking. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2024.
[49] Zirui Wang. Zihang Dai, B. Póezos, and Carbonell. Characterizing and avoiding negative transfer. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018
[50] Zirui Wang. Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual models: Findings and meta-learning trealment. In Conference on Empirical Methods in Natural Language Processing, 2020.
[51] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In IEEE International Conference on Image Processing, 2017.
[52] Xingjiao Wu, Luwei Xiao, Yixuan Sun. Junhang Zhang, Tianlong Ma, and Liangbo He. A survey of human-in-thee loop for machine learning. Future generations computer systems, 2021.
[53] Yihong Xu, Aljosa Osep, Yutong Ban, Radu Horaud, Laura Leal Taixé, and Xavier Alameda- Mlaneda-Pineda. How to train your deep multi-object tracker. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2020.
[54] Enneng Yang. Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Mmss, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.00 07666666..666... 2024.
[55] En Yu, Songtao Liu, Zhuoling Li, Jinrong Yang. Zeming Li, Shoudong Han, and Wenbing Tao, Generalizing multiple object tracking to unseen domains by introducing natural language representation. In Proceedings of the AAAI Conference Artificial Intelligence, 2023.
[56] En Yu, Tiancai Wang, Zhuoling Li, Yuang Zhang, Xiangyu Zhang, and Wenbing Tao. Motrv3; Release- fetch supervision for end-to-end objei- tracking. arXiv preprint 2305 2023.
13

---

[57] Jiazuo Yu. Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-expens adapters. In Proceedings the IEEE conference on Computer Vixion and Puttern Recognition, 2024.
[58] Fangao Zeng, Bin Dong. Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. Proceedings of European Conference on Computer Vision, 2022.
[59] Jianyu Zhang and Bottou, Learning useful representations for shifting tasks and distributions. Interna- tional Conference on Machine Learning, 2022.
[60] Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arithmetic operation. Advances in Neurul Information Processing Systems, 2023.
[61] Yifu Zhang, Peize Sun, Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In Proceedings of the European Conference Computer Vision, 2022.
[62] Yifu Zhang, Chunyu Wang. Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairness of detection and re-identification in multiple object tracking. International Journal of Computer Vision, 2021.
[63] Yuang Zhang. Tiancai Wang. and Xiangyu Zhang. Motrv:: Bootstrapping end-to-end multi-object tracking by pretrained object detectors. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, 2023.
[64] Zangwei Zheng, Mingyu Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision- iisoon-language models. IEEE International Conference Computer Vision, 2023.
[65] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. In Proceedings of the European Conference on Computer Vision, 2020.
14

---

Appendix
A Extracting task vectors from scale & shift layers
our convolutional backbone, we apply a channel-wise scale and shift 2 operation to adapt each backbone layer. We provide details computing the task vector as increment relative to the base parameters. Following (23), this is achieved by re-parameterizing
scale and shift as:
F EmER(s) Am)mm)
(Wo * +b) + (EmER()) Am.mms
(A)
EmeR(i) Am 7m & Wo) *h + EmER(s) Am (7m O bo + 9m
(B)
W
where Wo Cout X and b0 Cout represent the original convolution weights and bias, Cout is the number of convolution output channels, Cin is the number of convolution input channels. H and W are the height and width of the kernel, h is the output the preceding layer, and denotes the convolution operation. Notice that, to simplify the notation, we assume to reshape I'm E 1oCoxxxxxx and implicitly broadcast accordance with the dimensions of Wo before applying the Hadamard product @.
The task vectors T.y W
Wo and Tg b²- - bo for the scale and shift parameters defined:
Ty E EmER(() Am 7m 2 Wo) Wo:
(C)
Ta EmeR(s) Am (Tm bu Bm)) bp-
(D)
By representing our attributes as task vectors and leveraging pre-computed weights, we ensure that the inference process incurs no additional computational costs.
B Dataset licenses
MOTSynth released under the MIT License.
MOT17 is released under the CC BY-NC-SA 3.0 License.
PersonPath221 is released under the CC BY-NC 4.0 License.
C Dataset statistics
Tab. A presents statistics on the employed datasets, detailing attributes at both per-sequence and per-frame levels. We manually annotated these attributes, developing custom annotation tool that displays the first frame of each sequence and allows for efficient annotation using keybindings. This process required minimal effort, involving annotator for approximately three hours MOTSynth and two hours on PersonPath22. As shown in Tab. A the statistics indicate an imbalance certain attributes; address this, we implemented custom training sampler ensure that each module receives an equal number of backward iterations.
D Forgetting on source dataset
Recently, there has been growing interest Continual Learning for large pre-trained models, particu- larly in incrementally fine-tuning these models using parameter-effiiiem methods [4 57 A key challenge in this process avoiding the issue of catastrophic forgetting [30], where model loses knowledge from earlier training new tasks are introduced. To this end, we evaluated the extent of forgetting in the model when using task- specifce modules versus training the entire model. Namely, we start from the PASTA and MOTRv2-MS trained on MOTSynth as in Tab. [ Then, we further fine-tune such models on MOT17 and evaluate again on MOTSynth to measure the source-domain performance after the adaptation. As shown in Tab. B the modular approach trained MOT17 less prone to forget pre-training on MOTSynth, achieving superior results compared full fine-tuning when tested again MOTSythh test split. Specifically, modular approach PASTA
15

---

Table A: Per-sequence and per-frame attributes statistics in PersonPath22, MOT17, and MOTSYyth.
PersonPath22
MOT17
MOTSynth
Per-sequence altributes
Total Sequences
236
12
764
Indoor
61
57
Outdoor
175
12
707
Camera Low
162
10
479
Camera Mid
68
4
244
Camera High
6
41
Moving
60
8
220
Static
176
6
544
Bad Light
23
0
189
Good Light
213
14
575
Per-frame attributes
Total Frames
203653
5316
1375200
Occupancy Low
44%
24%
29%
Occupancy Mid
53%
54%
68%
Occupancy High
3%
22%
3%
Table Source-domall (MOTSynth) results before and after fine-tuning on target-domain (MOT17).
We report the difference in performance in brackets.
MOTSynth
HOTA1
IDFI
MOTA
DetAf
AssA
Trained on MOTSynth (Tab. 7
MOTRx2-MS
52.4
56.6
61.9
56.4
49.0
PASTA
53.0
57.6
62.0
56.2
50.4
Subsequently trained on MOT17
MOTRv2-MS
48.1 (-4.3)
56.3 (-0.3)
60.8 (-1.1)
50.7 (+5.7)
46.2 (-2.8)
PASTA
49.8
(-3.2)
57.4 (-0.2)
61.8 (-0.2)
52.3 (-3.9)
48.0 (-2.4)
outperforms the standard one across all metrics, demonstrating the effectiveness of the modular training in mitigating catastrophic forgetting. Indeed, the LoRA (16) modules act as regularizer that mitigates forgetting of the source-domain
E
Additional ablation studies
To further support our claim on leveraging all modules in zero-shot scenario, we conduct an additional ablation study on the test split of PersonPath22. As shown Tab. [ this test confirms that retaining knowledge from modules not directly related to the specific scenario is beneficial when dealing with domain shifts. Specifically, our selection strategy outperforms the unweighted average by empirically assigning weight 0.8 to the selected modules and p 0.2 to the others.
Comparison with tracking-by-detectin by-dttcciion. To comprehensively evaluate our method, we herein test BrteTrack and other bocking-by-detectin methods in zero-shot setting from MOT17 to PersonPath22. We report the results on PersonPath22 in Tab. D Results indicate that PASTA leads remarkable improvements compared the other query-based end-to-end approach (i.e.. methods (such
MOTRv2 (63)). even though they are both outperformed by the as BrteTraakk (1 To be more comprehensive, PASTA remains competitive terms of association performance (IDF1), but it yields weaker detection capabilities. Such trend does not surprise and line with what occurs in the more standard evaluation, where mue-tuning on the target dataset allowed. Indeed, beokiing-b-deeectte approaches are generally more robust than those based
16

---

Table C: Ablation study on different module aggregation strategies on PersonPath22 test set in zero-shot.
PersonPath22
MOTA
IDFI
FP
FNL
IDSW
Sum (only selected)
0.91
0.64
Avg. (only selected)
49.6
53.6
1821
105 611
6321
Weighted avg. (all)
50.0
53.8
17 786
105 454
6037
Table D: Comparison with tracking-by-detecinn approaches in zero-shot setting from MOT17 to PersonPath22.
Setting
IDF1
MOTAf
FP4
FNL
IDSW
Byte'Track
fine-tuned on PP22
66.8
75.4
17214
40
5931
ByteTrrack
zero-shot
56.2
55.9
3307
106 892
3962
OC-SORT
zero-shot
55.6
59.9
3254
94 786
5786
SORT
zero-shot
48.5
57.4
40 173
56
14.060
MOTRv2-MS
zero-shot 43.9
51.5
8304
119
5342
PASTA
zero-shot 46.1
54.6
7895
114
4702
end-to-end learning, so much so that is an established practice to present the results in separate to deliver apple-to-apple comparison.
parts of table 2 58 63),
In a zero-shot setting, we conclude that existing tracking- baee------- by-detection trackers are more robust to domain shifts. In these approaches, the only component potentially subject to shifts is the detector (e.g., YOLOX [4].. Instead, the motion model (e.g., Kalman Filter [9] and the association almost parameter-free procedures that less affected by domaín shifts for construction, as their design reflects strong inductive biases about human motion. For such reason, belief that the problem of domain shift in Multiple Object Tracking (MOT) should be primarily addressed in parametric approaches such as deep neural networks. For this reason, our research question focuses query-based trackers (e.g., MOTRv2) that learn entirely from data. Our final goal is to enhance these trackers, as their end-to-end nature results can lead to challenges during domain shifts.
strategy 3 61
Byte Track thresholds. In Tab. [ we evaluated Byte'rrack on MOTSynth using the default input- parameters provided in the public ByteTraak repository, specifically minimum confidence score (min_scorv score) of 0.1 and a track threshold (track_ thresh) of 0.6. In Tab. E we present the results for different values of these thresholds. The results close, with slight improvement when reducing the track_thress to 0.3 or 0.4, while min_score of 0.1 remains optimal.
F
On computational costs
Memory efficiency. We compare the GPU memory of full fine-tuning versus our approach on the MOTSynth dataset. Our method reduces training GPU memory requirements from 13GB to 8.25GB (for batch size of I), reduction of over 35%. This significant decrease is due to the lower number of parameters updated by the optimizer: 42M parameters for standard fine-tuning versus 15M for our PEFT technique, reported in Tab.
Inference speed. Additionally, our approach does not add any overhead during inference, aside from weight merging, which is negligible for stationary attributes, compared to MOTRv2, which maintains speed 6.9 FPS on 2080Ti GPU.
Storage efficiency. Using PEFT techniques significantly reduces storage needs. Without these techniques, each attribute would require fully fine-tuned model, which poses several challenges, especially in memory constraints [16] 25 Firstly, storing separate model for each attribute is highly storage-intensive. For instance, PASTA module is approximately 5MB, whereas the full model exceeds 350MB. With 12 attributes, the total storage requirement for PASTA would be 410MB
17

---

Table ByteTrrack thresholds sensitivity analysis on MOTSynth.
Score
Threshold
HOTAT
IDFIt
MOTAT
DetAf
AssAt
0.1
0.2
45.9
56.4
61.9
51.1
41.6
0.1
0.3
46.0
56.5
62.1
51.1
41.7
0.1
0.4
45.9
56.6
62.2
50.9
41.8
BrteTrack
0.05
0.6
43.0
54.4
54.5
45.9
40.9
0.1
0.6
45.7
56.4
61.8
50.1
41.9
0.2
0.6
45.6
56.3
61.5
49.8
41.9
0.1
0.7
45.0
55.8
60.3
48.7
41.8
PASTA (Ours)
53.0
57.6
62.0
56.2
50.4
(350MB + 12 x 5MB). In contrast, storing 12 fully fine-tuned models would require around 4.2GB (12 350MB), representing tenfold increase in storage needs. Additionally, adapting entire model to each specific condition is more time-consuming than using LoRA, as involves optimizing more significant number of parameters. This adaptation process must be repeated for each attribute, making it both impractical and costly. Moreover, fully fine-tuning a transformer-base architecture demands more data than parameter-efficient approach.
G Limitations
One limitation of our approach is the reliance on expert router, which requires manual data annotation or intervention by an external domain expert. This process can be resource-intensive and may not scale well for larger datasets or diverse scenarios. Future work may explore the development of automatic routing techniques, which could significantly improve scalability, performance, and ease of deployment in real-world applications by reducing the dependency manual annotations.
H Societal impacts
Positive Impacts. Enhanced security and surveillance is one of the key benefits of this work. Improved accuracy and robustness in tracking can lead to better crime prevention, more efficient law enforcement, and increased public safety. Additionally, operational efficiency is another positive impact, where various sectors, including transportation, retail, and urban planning, can benefit from optimized operations and resource allocation. Moreover, customization and adaptability are enhanced by tailoring modules for specific scenarios, increasing versatility in applications ranging from healthcare to sports analytics.
Negative Impacts. However, there also potential negative impacts to consider. Privacy concerns arise from increased tracking capabilities, which may lead unauthorized surveillance and privacy infringement. Bias and fairness are also issues, biased training data can perpetuate existing biases, leading to unfair treatment of certain groups
While the modular approach presents significant advancements, is crucial to address these societal impacts through careful design and transparent policies.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction sections clearly articulate the main contributions and scope of the paper, providing accurate overview of its objectives and the significance its findings.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Appendix G discusses the limitation of relying on expert knowledge, specif- ically using a static router for the attributes. This approach necessitates annotating each attribute within the dataset, which can be time-consuming.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The assumptions made in this paper are supported by the experiments provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification:
The paper deeply discusses the implementation details in Sec. 5.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We release models and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
gublic/guidess httpi///uuiiee//eeeubbbiieeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips ccpps.cc/ppp
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper discusses the training and test details necessary to understand the results S.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not reported in results due to the computational expense that would have been incurred in generating them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: The paper reported the GPU and iteration needed for training our approach
in Sec. 53
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httiiis/httt::///t/
Answer: [Yes]
Justification:
The research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts reported in appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the original authors have been properly cited, and the relevant licenses reported in the appendix.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Eg20Akl9oV", "reviewer": "Reviewer_nupV", "summary": "The paper presents a multiple object tracking framework that can generalize to new domains by training specialized modules for each scenario attributes.\nThese modules are trained using Parameter-Efficient Fine-tuning and modular deep learning techniques on a transformer-based tracker. \nThis tracker is build on the Deformable DETR framework with a ResNet backbone for image feature extraction.\nExtensive experiments are performed on both synthetic and real datasets to compare with some recent track-by-detection and track-by-attention approaches.\nIt shows that the proposed approach can generalize well on unseen datasets and reduce negative interference during training.", "questions": "1. Is it necessary to have training data for all possible cases to train all the parameter modules?\n2. How does the model trained on the MOT17 training set perform on the PersonPath22 test set? This will also demonstrate the generalization ability of the model in the zero-shot setting on a new domain.\n3. It is not clear why MOTRv2 is not compared in the fully-trained section in Tables 2 and 3 while TrackFormer [24] is included in both fully-trained and zero-shot.", "limitations": "Yes. The authors provided some analysis on limitations and social impacts.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed modular architecture is significant and origin in solving MOT problem.\n2. Experiments support and prove the proposed architecture's ability to generalize on new domains, i.e. real dataset from synthetic dataset.", "weaknesses": "1. The authors only conduct a synth-to-real zero-shot experiment. It would be better to see real-to-real experiment where one can train on one dataset and test on a new dataset.\n2. It would be great to have some statistic on the training dataset in terms of the attributes, e.g. how many frames / videos have high/low occupancy, etc. \n3. One missing aspect in the ablation studies is how the task vector is incorporate to the pre-trained tracker contribute to the final results. For examples, incorporating on the backbone, encoder or decoder only and the whole network can have different results.\n4. The proposed approach seems to be limited by the availibility of the attribute data for training each scenario-specific module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable questions and for having appreciated the originality and efficacy of the proposed approach.\n\n**W2 - Dataset statistics**\n\nWe report the requested details in the following tables, which provide statistics on the employed datasets divided by per-sequence and per-frame attributes\n\n| Per-sequence attributes | PP22    | MOT17   | MOTSYNTH |\n| -- | ------- | ------- | -------- |\n| **Total Sequences**     | **236** | **12**  | **764**  |\n| Indoor| 61      | 2       | 57       |\n| Outdoor| 175     | 12      | 707      |\n| Camera Low| 162     | 10      | 479      |\n| Camera Mid| 68      | 4       | 244      |\n| Camera High| 6       | 0       | 41       |\n| Moving| 60      | 8       | 220      |\n| Static| 176     | 6       | 544      |\n| Bad Light| 23      | 0       | 189      |\n| Good Light| 213     | 14      | 575      |\n\n\n\n| Per-frame attributes | PP22   | MOT17 | MOTSYNTH |\n| -- | ------ | ----- | -------- |\n| **Total Frames**     | **203653** | **5316**  | **1375200**  |\n| Occupancy Low        | 44%    | 24%   | 29%      |\n| Occupancy Mid        | 53%    | 54%   | 68%      |\n| Occupancy High       | 3%     | 22%   | 3%       |\n\n**W3 - Ablation of task vectors contribution**\n\nWe agree that an additional ablation study on where to apply the task vectors might clarify which components are most critical for adaptation over scenario attributes. To investigate this aspect, we trained three versions of PASTA by omitting the task vectors from the encoder, decoder, or backbone, respectively. We will report these results computed on MOTSynth in the final manuscript:\n\n|| HOTA | IDF1 | MOTA | DetA | AssA |\n|----|------|------|------|------|------|\n| MOTRv2-MS| 52.4 | 56.5 | 61.9 | 56.4 | 49.0 |\n| PASTA - no decoder| 51.5 | 56.0 | 58.9 | 53.6 | 49.8 |\n| PASTA - no encoder| 52.4 | 56.9 | 61.2 | 55.7 | 49.7 |\n| PASTA - no backbone| 52.5 | 57.0 | 61.5 | 55.6 | 49.9 |\n| PASTA - all| 53.0 | 57.6 | 62.0 | 56.2 | 50.4 |\n\nThe results indicate that not applying task vectors to the decoder significantly degrades detection and association metrics. We believe that this degradation can be explained by considering the crucial role of the decoder. The decoder must indeed gather information from detection, tracking, and proposal queries while simultaneously integrating visual information from the encoder. Consequently, not adapting the decoder prevents the architecture from effectively leveraging queries and visual cues. The encoder also contributes substantially, though to a lesser extent than the decoder, as it primarily refines and contextualizes visual features from the backbone. The backbone shows the smallest contribution.\n\n**Q1/W4 - Attributes availability during training**\n\nDuring training, it is important to identify the relevant attributes for the specific problem at hand. In our approach, the selection of attributes was primarily guided by common sense and our experience with the task. Also, we chose attributes that are generally applicable to most use cases. However, this selection serves as a proof of concept. In practical applications, the selection could be further refined to reflect specific characteristics of the problem under consideration.\n\nFor instance, in a surveillance scenario characterized by indoor cameras, the indoor/outdoor attribute may become unnecessary. Conversely, an attribute for weather conditions might be valuable in locations with variable weather. \n\nThe need for predefined attributes is easily addressable using automatic or semi-automatic classifiers. For example, an analysis of the brightness level can easily classify lighting conditions or a straightforward detector can be exploited to count objects of interest in the scene, thereby classifying crowd density.\n\n**W1/Q2 - Real-to-real (MOT17 → PersonPath22)**\n\nWe report below the results of the model trained on MOT17 and evaluated in zero-shot on PersonPath22.\n\n| MOT17->PP22 | HOTA | IDF1 | MOTA | FP |   FN   | IDsw |\n|-------------|------|------|------|----|--------|------|\n| MOTRv2      | 43.9 | 51.5 | 48.6 |8304| 119391 | 5342 |\n| PASTA       | 46.1 | 54.6 | 50.8 |7895| 114620 | 4702 |\n\nTo provide a comparison, we train MOTRv2 on MOT17 and test their performance on PersonPath22. Our approach yields better results compared to fine-tuned MOTRv2, demonstrating that exploiting modules improves the model's generalization capabilities on new and/or real domains. We will include these findings in the final revision of the manuscript.\n\n**Q3 - Missing MOTRv2 full tuning in Tables 2 - 3**\n\nWe agree that including fine-tuned MOTRv2 results would provide a better understanding of the zero-shot experiments. We primarily did not undertake it due to time constraints and computational overheads. Herein, we remedy this and report the results for Table 2 (MOT17):\n\n|| HOTA | IDF1 | MOTA | DETA | ASSA |\n| ------------------------- | ---- | ---- | ---- | ---- | ---- |\n| MOTRv2 (Fine-tuned by us) | 66.8 | 78.9 | 73.2 | 62.5 | 71.4 |\n| MOTRv2-MS (zero-shot)     | 62.6 | 73.0 | 67.6 | 60.3 | 65.5 |\n| PASTA (zero-shot)         | 64.0 | 74.9 | 68.1 | 60.4 | 68.3 |\n\nNotably, our zero-shot approach performs similarly to its fine-tuned counterpart. We appreciate the reviewer highlighting this missing comparison, which strengthens our findings.\n\nRegarding Table 3 (PersonPath22), we have included fully trained methods on PersonPath22 as reported in the PersonPath22 paper. We could not train MOTRv2 on this dataset primarily because the authors of the PersonPath22 did not release the YOLOX weights they used for the other methods listed in their table nor provide interpolation scripts for the training-set annotations. Since training MOTRv2 on PersonPath22 is not crucial for the PASTA evaluation, and given the abovementioned limitations, we opted to refrain from reporting the full tuning of MOTRv2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "33I8MOq0d4", "reviewer": "Reviewer_oFC1", "summary": "The paper introduces a fine-tuning framework, denoted PASTA, for multiple-object tracking that is aimed at reducing the cost of tine-tuning large models while mitigating negative inference to improve zero-shot transfer and domain generalization.\n\nDuring training, the authors independently fine-tune per-domain modules/experts on pre-defined scenario attributes (e.g. lighting, static or moving camera, ...). \nModules only adapt a sub-set of the hyperparameters and the adaptation is done by relying on low-rank adaptation (LoRA). This allows for fine-tuning with a low computational and memory footprint.\nMore specifically, the authors only adapt batch norm parameters in the backbone and linear layers in the transformer modules.\n\nDuring inference, the method relies on \"expert knowledge\" to select the right module for each attribute (e.g. select the \"bad lighting\" module for the \"lighting\" attribute). A convex combination of the attribute's modules' weights is then applied (with the selected module having a higher coefficient, defined by a hyper-parameter).\n\nFor the zero-shot (transfer) setting, the model is simply set to be a weighted average of all modules.\n\nExperiments demonstrate the effectiveness of the method by comparing performance with the same model trained without the framework. PASTA outperforms the baseline on both the synthetic dataset and on zero-shot transfer (from synthetic to real-world datasets). The authors further demonstrate the reduced forgetting in the ablation studies.", "questions": "- What is the computational gain in terms of time and memory in comparison to the full fine-tuning of MOTRv2?\n- How does the method relate to domain adaptation and zero-shot methods?", "limitations": "Yes, the authors discuss the potential positive and negative societal impacts of multiple-object tracking, as well as the main limitation of the method (i.e. the reliance on an external \"domain expert\")", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is easy to read, provides a good overview of the methods it relies on (i.e. LoRA and modular deep learning), and includes a good overview illustration.\n- The experimental section and the ablation studies demonstrate the effectiveness of the approach and thereby validate the main claims.\n- The authors discuss both positive and negative possible societal impacts of the approach.\n- \"Novelty\" is limited as it is mainly an implementation of existing concepts. Nevertheless, the application to a new domain and the good execution make it a useful contribution to the community.", "weaknesses": "- The approach is claimed to reduce fine-tuning costs (reduced time and memory footprint), but no time or memory comparison to full tine-tuning is provided.\n- The approach is related to domain adaptation and zero-shot transfer, but does not discuss similarities and differences to those approaches in the related work section, and does not compare to such methods in the experimental section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "As pointed out by the reviewer, a more detailed discussion about the computational costs and the relationship with other domain adaptation methods would enhance the quality of our manuscript. Below, we answer these questions and will integrate this information into the final revision.\n\n**W1/Q1 On the computational cost**\n\nWe compare the GPU memory and time requirements of full fine-tuning versus our approach on the MOTSynth dataset.\n\nOur method reduces training GPU memory requirements from 13GB to 8.25GB (for a batch size of 1), a reduction of over 35%. This significant decrease is due to the lower number of parameters updated by the optimizer: 42M parameters for standard fine-tuning versus 15M for our PEFT technique. Regarding timing, we achieved a reduction from 0.6 seconds per iteration to 0.55 seconds per iteration on average, which is approximately a 10% improvement. This is because most of the network parameters remain frozen. Overall training time is reduced from around four days to about 3.5 days on average.\n\nAdditionally, since our final model is easily adaptable to a wide variety of scenarios without further fine-tuning, the overall time requirement for deployment across diverse domains is sensibly less compared to standard fine-tuning.\n\nFinally, our approach does not add any overhead during inference, aside from weight merging, which is negligible for stationary attributes, compared to MOTRv2, which maintains a speed of 6.9 FPS on a 2080Ti GPU.\n\n**W2/Q2 Relation with domain adaptation and zero-shot methods**\n\nWe acknowledge that the number of works handling domain adaptation and MOT or zero-shot MOT is very limited. We will make a comparison of our approach with the most relatable in the following:\n\n### **Domain Adaptation methods**\n\nTo the best of our knowledge, no tracking-by-query method currently employs domain adaptation. However, we have identified two tracking-by-detection methods that utilize domain adaptation: Simple Cues Lead to a Strong Multi-Object Tracker (GHOST) [30] and DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [29].\n\n**GHOST** is a simple tracker that leverages detections from an off-the-shelf detector. Specifically, GHOST employs \"on-the-fly Domain Adaptation,\" which involves updating the Batch Norm layer statistics during inference (similar to test-time adaptation techniques) in the layers handling reID features. There are several critical differences between GHOST and our approach. First, we only combine already trained modules. Second, the adaptation in GHOST is applied only to the reID module features, whereas our adaptation affects the entire network. Since our approach is end-to-end and query-based, it impacts the tracking and detection components.\n\n**DARTH** employs a test-time adaptation (TTA) technique to adapt the model from a source dataset to a target one using a Knowledge Distillation approach. Each adaptation step requires three forward passes to compute all objective functions, making the process computationally and memory intensive. Additionally, DARTH performs offline TTA, adapting the model using the entire sequence before evaluating it, which is not comparable to our fully online approach. A key difference from our method is that DARTH's requirement of having the entire sequence available for adaptation makes it challenging for real-world use cases. In contrast, our approach only requires simple attributes of the target scene, eliminating the need for further training or adaptation during deployment.\n\n### **Open-vocabulary methods**\nRecent advancements in zero-shot multiple object tracking demonstrate a significant shift towards open-vocabulary multiple object tracking, a specific setting where textual descriptions define new categories to track. In this context, we recognize the latest works. **OVTrack** is an open-vocabulary tracker based on Faster R-CNN, enhanced with CLIP distillation to learn open-vocabulary capabilities. It also employs a hallucination strategy using denoising diffusion models for robust appearance feature learning. **Z-GMOT** introduces a dataset comprising videos and textual descriptions of target attributes and a tracker that improves detection through advanced grounded language-image pretraining. **OVTracktor** is another open-vocabulary tracker that can detect and segment any category.\n\nOur method, however, does not involve open-vocabulary and language models. Specifically, PASTA focuses on comprehensively transferring domain knowledge of end-to-end trackers with a modular approach. Unfortunately, such open-vocabulary methods are not easily comparable to our approach. They require additional text data during inference and produce results only on open-vocabulary datasets rather than classical MOT benchmarks like MOT17.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TQ2vlxcyxm", "reviewer": "Reviewer_6tD5", "summary": "This paper aims to address the domain gap across different multi-object tracking datasets. The proposed method is inspired by LoRA and introduces parameter-efficient fine-tuning for state-of-art end-to-end trackers. Specifically, these trackers are first trained on a large-scale dataset MOTSynth. After that, by shifting several expert attributes and training partial parameters, these trackers can achieve impressive performance on MOT17 and PersonPath22.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The proposed method is novel and interesting. I like it very much!\n3. The experiment details are well provided.", "weaknesses": "The idea of this paper is attractive for me. However, I still have several minor concerns about the experiments:\n1.This paper does not perform zero-shot learning on another popularly used dataset MOT20.\n2.The employed attributes in this work, such as lighting, viewpoint, occupancy, location, and camera, seem “weak”. How about the transferring performance on “strong” attributes, like scenes and categories? In other words, how about the zero-shot evaluation performance on other datasets like DanceTrack and KITTI?\n3.In Table 1, both model parameters and trainable parameters should be listed for a better comparison. Besides, the table captions in experiments are suggested to provide more descriptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the novelty and efficacy of our proposed approach. We will answer in the following the doubts of the reviewer:\n\n**W1 - Evaluation on MOT20**\n\nWe decided not to evaluate our method on MOT20 since its variety is limited in terms of attributes, as it contains only four similar sequences specifically designed to be crowded. In contrast, the benchmarks we use—MOT17, MOTSynth, and PersonPath22—capture a wide range of attributes and scenarios. The lack of diversity implies that MOT20 as a benchmark might not be representative for assessing our attribute-based approach. For instance, MOT20 lacks scenarios with low or moving cameras. We report in the following the complete MOT20 attribute statistics.\n\n| Per-sequence attributes  | MOT20-TRAIN | MOT20-TEST |\n| ------------------------ | ----------- | ---------- |\n| **Total Sequences**      | **4**       | **4**      |\n| Indoor                   | 2           | 1          |\n| Outdoor                  | 2           | 3          |\n| Camera Low               | 0           | 0          |\n| Camera Mid               | 2           | 3          |\n| Camera High              | 2           | 1          |\n| Moving                   | 0           | 0          |\n| Static                   | 4           | 4          |\n| Bad Light                | 1           | 1          |\n| Good Light               | 3           | 3          |\n\n**W2 - On scenes as attributes**\n\nWe chose \"weak\" (fine-grained) attributes to better align with fine-tuning approaches. However, it would be interesting to explore the addition of \"strong\" (coarse) attributes to represent completely different scenes or settings. For example, an attribute for automotive settings and another for pedestrian settings. We feel that the rebuttal phase is too short to do justice to such an evaluation, which requires extensive training and experiments. However, we greatly appreciate the reviewer’s suggestion and aim to carry it out in future work.\n\n**W3 - Table 1 parameters**\n\nIn Table 1, we reported the total number of trainable parameters (15M for PASTA vs 42M for MOTRv2). Once trained, the total number of parameters in our approach is the same as MOTRv2, i.e., 42M. We will provide a clearer specification of the number of parameters in the final version of the manuscript and will include a better description in the table captions. We thank the reviewer for this suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jAVTHEYqII", "reviewer": "Reviewer_4fn5", "summary": "This paper proposes a new framework called PASTA (Parameter-Efficient Scenario-specific Tracking Architecture), which aims to improve the generalization ability of multi-object tracking (MOT) in diverse scenarios. The main contributions of this paper include: 1) proposing the PASTA framework to achieve efficient query tracker fine-tuning through PEFT technology; 2) improving domain transfer and preventing negative interference by introducing expert modules; 3) validating the method's effectiveness in zero-shot tracking scenarios through a comprehensive evaluation.", "questions": "1，The modular design proposed in the paper effectively avoids negative interference and improves domain generalization ability. However, the selection and combination of modules can be further optimized using more intelligent methods, such as reinforcement learning or other automated techniques, instead of relying on manual selection by domain experts. Intelligent optimization of module selection can further enhance the model's adaptability in complex scenarios while reducing dependence on domain experts, thereby increasing the practical application value of the system.\n2，The paper verifies the effectiveness of PASTA on multiple datasets, but it should consider validation in more real-world application scenarios, such as traffic monitoring, smart retail, autonomous driving, etc. These fields have unique tracking needs and challenges. By verifying PASTA in these scenarios, the wide applicability and practical value of PASTA can be further demonstrated, enhancing the paper's persuasiveness and showcasing PASTA's potential application effects in different fields.\n\n3， When combining modules, the synergy between modules should be considered, and related experiments should be added to explore the performance impact of different module combinations. Specifically, experiments can be designed to evaluate which module combinations can produce the best effects or which combinations may lead to performance degradation. In-depth research on module synergy can help understand and optimize the PASTA module design, improve overall model performance, and provide valuable references for future module development.\n4，Discussion of Computational Resources and Time Costs： Although PASTA reduces computational costs through parameter-efficient fine-tuning, the specific computational resources and time costs in practical applications have not been discussed in detail. Particularly in comparison with traditional methods, clearly listing the specific computational resources (such as GPU/CPU time, memory requirements, etc.) and time costs required in the training and inference stages can help readers better understand the feasibility and advantages of PASTA in practical applications. This analysis is crucial for evaluating the practical benefits of PASTA in large-scale real-world applications.\n5，The discussion of future work and application prospects of the PASTA framework is relatively brief and does not fully demonstrate its long-term value and expansion potential.", "limitations": "1，Complexity of Modular Systems\nThe modular design of the PASTA framework increases the complexity of the system, especially in terms of training, managing, and combining modules. Each module requires separate training, optimization, and validation, increasing development and maintenance costs. In practical deployment, modular systems require more complex architectural support, involving dynamic selection and combination of modules, which may increase implementation difficulty and system maintenance complexity.\n2，Generalization Limitations\nAlthough PASTA performs well on specific datasets and scenarios, its design and optimization target specific scenario attributes (such as camera angles, lighting conditions, etc.). PASTA's generalization and adaptability may be limited when dealing with new scenarios that do not belong to these predefined attributes. The framework's performance in completely different application environments or new domains has not been fully validated, potentially affecting its promotion and widespread application.\n3，Dataset Dependency\nThe experiments in the paper are mainly based on the MOTSynth, MOT17, and PersonPath22 datasets, which may have certain biases or characteristics. The excellent performance of the PASTA framework on these datasets may not be directly generalizable to other datasets or more diverse data sources. The lack of experimental validation on more diverse datasets limits the comprehensive evaluation of PASTA's generalization ability in various scenarios.\n4，Dependence on Pre-trained Models\nThe overall performance of the PASTA framework largely depends on pre-trained backbone networks and detectors. If pre-trained models perform poorly in certain tasks or scenarios, the performance of the PASTA framework will also be significantly affected. This means that the application and effectiveness of the PASTA framework are, to some extent, limited by the quality of pre-trained models, requiring high-quality pre-trained models to achieve optimal performance.\n5，Lack of Evaluation of Real-time Performance\nThe paper does not provide a detailed evaluation of the real-time performance of the PASTA framework, such as processing latency and efficiency in real-time video streams. Real-time performance is critical for many practical applications (such as video surveillance and autonomous driving). The lack of evaluation in this aspect makes it difficult to judge the feasibility and effectiveness of PASTA in real-time environments, potentially limiting its application prospects in scenarios requiring high real-time performance.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1，The paper thoroughly verifies the effectiveness of the PASTA framework through extensive experiments on multiple datasets (MOTSynth, MOT17, PersonPath22). These experimental results clearly demonstrate the significant advantages of PASTA in reducing negative interference and improving domain generalization ability, particularly its excellent performance in zero-shot settings. On the MOTSynth test set, PASTA shows improvements in multiple key metrics (such as HOTA, IDF1, MOTA, DetA, AssA) compared to MOTRv2-MS, proving the advantage of its modular design in handling complex scenarios.\n2，The paper is structured tightly and logically, with each part (introduction, methodology, experiments, and conclusion) clearly laid out, making it easy for readers to follow and understand. The methodology section, in particular, provides detailed descriptions of the design principles and implementation steps of the PASTA framework, allowing readers to clearly grasp its working principles and innovations.\n3，The paper uses professional and accurate academic language, clearly expressing the research objectives, methods, and results. The concise and clear description of technical details and experimental results enhances the paper's persuasiveness and credibility.\n4，The figures and tables in the paper are exquisitely designed, visually presenting experimental results and method structures. For example, Figure 1 shows the modular architecture of PASTA, allowing readers to intuitively understand the combination and application of different modules. The experimental result tables are also clear, facilitating the comparison of different methods' performances.\n5，The PASTA framework combines Parameter-Efficient Fine-Tuning (PEFT) and Modular Deep Learning (MDL) technologies, proposing a new multi-object tracking solution. By defining key scenario attributes and training specialized PEFT modules for each attribute, PASTA performs excellently in handling heterogeneous scenarios, demonstrating significant innovation and research value. The paper introduces the concept of combining modules using task arithmetic, significantly reducing negative interference and enhancing domain generalization ability, showcasing the potential of modular design in deep learning.", "weaknesses": "1，The PASTA framework relies on manual selection by domain experts to determine the appropriate modules for the current scenario. Although effective, this method has limitations in practical applications. In some scenarios, it may not always be possible to obtain support from domain experts, or the judgments of domain experts may be influenced by subjective factors, affecting the overall performance of the model. Additionally, this manual selection method may lead to consistency and repeatability issues and is challenging to automate in large-scale applications.\n2，Although the PASTA framework has been extensively tested on the MOTSynth, MOT17, and PersonPath22 datasets, these experiments mainly focus on pedestrian tracking and surveillance scenarios. There is a lack of validation in other important application fields (such as traffic monitoring, smart retail, autonomous driving, etc.), limiting the generality and promotion of the results. Further experiments can help verify the application effects of PASTA in broader scenarios, enhancing its generality and practical value.\n3，The paper mentions that PASTA reduces computational costs through parameter-efficient fine-tuning, but it lacks detailed analysis of specific computational resources and time costs. For example, in practical applications, how much computational resources (such as GPU/CPU time, memory requirements, etc.) are specifically needed, and how much time is saved compared to traditional methods. Detailed analysis of resources and time costs can help evaluate the practical feasibility of PASTA in different application scenarios.\n4，Although the paper mentions the advantages of modular design, it lacks an in-depth study of the potential synergy effects between different module combinations. There may be complementary or conflicting effects between different modules, which have significant impacts on the final performance. Conducting relevant experiments and analysis can help understand which module combinations are most effective, thereby optimizing the design and application strategies of PASTA, further enhancing model performance.\n5，The paper does not provide a detailed discussion on the robustness of model parameters in the PASTA framework. There is a lack of experiments testing the model's performance under different noise levels, data quality, and data volumes. These tests can verify the model's stability and robustness in practical applications, further improving the credibility and practicality of PASTA. These experiments can demonstrate the adaptability and stability of PASTA in various real-world environments, ensuring it maintains high performance under various conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1/Q1 - On the selection of modules**\n\nIn our work, we use a Domain Expert to select attributes, a practice that is not far from reality. For example, in fixed-camera scenarios, the mounting perspective is known and whether it will be indoors or outdoors. Lighting can be easily measured with a simple computer vision algorithm, and occupancy can be determined by counting detections. Therefore, the assumption of using a Domain Expert is both reasonable and practical.\n\n**W2/Q2/L2/L3 - On the evaluation on other domains**\n\nWe argue that MOT17 and PersonPath22 represent practical, real-world surveillance applications. Unfortunately, due to time constraints, we cannot perform other extensive further evaluations during the rebuttal phase.\n\n**W3/Q4 - On the computational cost**\n\nWe compare the GPU memory and time requirements of full fine-tuning versus our approach on the MOTSynth dataset. Our method reduces GPU memory requirements from 13GB to 8.25GB, a reduction of over 35%. This significant decrease is due to the lower number of parameters updated by the optimizer: 42M parameters for standard fine-tuning versus 15M for our approach. Regarding timing, we achieved a reduction from 0.6 seconds per iteration to 0.55 seconds per iteration on average, which is approximately a 10% improvement. Overall training time is reduced from around four days to about 3.5 days on average.\n\nAdditionally, our model's adaptability to various scenarios without fine-tuning significantly reduces deployment time across diverse domains.\n\n**W4/Q3 - Synergy between modules**\n\nWe kindly refer the reviewer to Tables 5 and 6 and Figure 2 for an analysis of the synergy between different modules. We explore various methods of aggregating and selecting modules. Selecting opposite modules (e.g., poor lighting when the scene has good lighting) results in worse performance than selecting the correct module. Specifically, Figure 2 demonstrates that adding the correct module one at a time monotonically increases both IDF1 and MOTA. We will make sure to highlight these experiments better in the final revision.\n\n**W5 - Module robustness**\n\nThe experiments we conducted on MOTSynth, MOT17, and PersonPath22 (Tables 1 to 3) cover a wide range of scenarios, attributes, data volumes, and quality. MOTSynth includes nearly 1.5 million frames and 800 different scenes, PersonPath22 encompasses over 200,000 frames and 236 scenes, while MOT17 comprises 5,316 frames across 14 scenes. These datasets differ significantly in volume and quality, with MOTSynth being a synthetic dataset featuring high-quality images and annotations. PersonPath22 also boasts good image quality and accurate annotations. Conversely, MOT17, having been around longer, presents more challenges due to its relatively lower image quality.\n\nAdditionally, our zero-shot experiments demonstrate PASTA's strong generalization capabilities and its resilience to noise, confirming its robustness across diverse conditions and datasets.\n\n**Q5 - Future Work**\n\nThe discussion about future work is indeed brief and could be expanded. One aspect that warrants detailed analysis is the routing technique. As suggested by the reviewer, employing reinforcement learning to select the most appropriate modules based on the current scene could be beneficial. Additionally, as suggested by 6tD5, expanding the modules to describe dataset-level characteristics, such as automotive versus pedestrian settings, would be an interesting avenue to explore. Another potential topic for future work involves applying additional domain adaptation techniques, such as GHOST and DARTH (as mentioned in response to oFC1), which could further enhance the adaptability of our approach. We will ensure that these future work discussions are thoroughly addressed in the final version of the manuscript.\n\n**L1 - Increase in complexity**\n\nOur framework introduces a set of parameters for each attribute, which is the only increase in complexity. Training and optimization of modules are conducted in parallel, closely aligning with a standard query-based tracker like MOTRv2. Validation requires only the merging of weights based on the scene to be evaluated. In the deployment phase, there is an additional requirement for selecting or weighting modules.\nThis slight increase in complexity offers significant benefits. Specifically, this additional step eliminates the need to fine-tune the model in specific environments. Our approach provides a highly adaptable and efficient tracker by customizing the attributes to suit different scenarios.\n\n**L4 - Dependance on pre-train**\n\nOur framework aims to enhance the knowledge transfer of end-to-end trackers. While the performance of the PASTA framework does depend on pre-trained backbone networks and detectors, we specifically address scenarios where these pre-trained models may perform poorly. Our approach involves fine-tuning on these specific scenarios without altering the pre-trained weights, as we only train a small, disjoint set of parameters. This allows us to retain the benefits of the pre-trained models while improving performance in targeted areas. In our experiments, we compare classical fine-tuning, named MOTRv2-MS, with our modular approach, showing better control and selection of parameters for different scenarios.\n\n**L5 - Real-time performance**\nOur approach does not introduce additional overhead compared to MOTRv2 during inference, so the frames per second (FPS) remain unchanged. Specifically, the YOLOX detector performs at 25 FPS, and MOTRv2 operates at 9.5 FPS with a 2080Ti GPU. When combining these two components, the overall speed is 6.9 FPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eJaC41YNcM", "reviewer": "Reviewer_Vx5C", "summary": "The paper introduces PASTA, a framework designed to address the challenges of training end-to-end transformer-based trackers in heterogeneous scenarios, specifically negative interference and poor domain generalization. PASTA leverages Parameter-Efficient Fine-Tuning (PEFT) and Modular Deep Learning (MDL) to define and train specialized modules based on key scenario attributes like camera viewpoint and lighting conditions. These modules are then combined using task arithmetic to enhance generalization to new domains. The framework's effectiveness is demonstrated through extensive experiments on MOTSynth and zero-shot evaluations on MOT17 and PersonPath22, where it outperforms traditional monolithic trackers. The primary contributions include the introduction of PASTA, the use of PEFT and MDL for better domain adaptation, and the validation of its superior performance in diverse tracking scenarios.", "questions": "I have listed my concerns and questions above.", "limitations": "The authors have claimed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces an innovative framework that combines Parameter-Efficient Fine-Tuning (PEFT) and Modular Deep Learning (MDL) to tackle challenges faced by end-to-end transformer-based trackers in heterogeneous scenarios, setting it apart from traditional approaches. The research quality is robust, with extensive experiments on MOTSynth, MOT17, and PersonPath22 validating the framework's effectiveness. The paper is well-structured and clearly explains key concepts and methodologies, making it accessible and easily understood.", "weaknesses": "The paper has several weaknesses that should be addressed. \n1. The authors do not compare their framework to ByteTrack and other trackers in zero-shot settings, limiting their evaluation's comprehensiveness. \n2. For a fair comparison in Table 1, the authors should provide detailed information on the different detection thresholds used, as varying thresholds can significantly affect ByteTrack's performance. \n3. The authors need to discuss their choice of attributes in more detail, explaining why specific attributes were selected and their relevance to the framework. \n4. It remains unclear if the performance could be improved without using LoRA, as the authors do not explore or report the impact of omitting LoRA in their experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 - Comparison with other methods in zero-shot**\n\nWe herein test ByteTrack and other tracking-by-detection methods in a zero-shot setting from MOT17 to PersonPath22. The results on PersonPath22 are as follows:\n\n|Tracker|Setting|IDF1|MOTA|FP|FN|IDsw|\n|-|-|-|-|-|-|-|\n|ByteTrack|fine tuned on PersonPath22|66.8|75.4|17214|40902|5931|\n|ByteTrack|zero shot results|56.2|55.9|3307|106892|3962|\n|OC-SORT|zero shot results|55.6|59.9|3254|94786|5786|\n|SORT|zero shot results|48.5|57.4|40173|56003|14060|\n|MOTRv2|zero shot results|51.5|48.6|8304|119391|5342|\n|PASTA|zero shot results|54.6|50.8|7895|114620|4702|\n\nResults indicate that PASTA leads to remarkable improvements compared to the other query-based end-to-end approach (i.e., MOTRv2), even though they are both outperformed by the tracking-by-detection methods (such as ByteTrack). To be more comprehensive, PASTA remains competitive in terms of association performance (IDF1), but it yields weaker detection capabilities. Such a trend does not surprise us and is in line with what occurs in the more standard evaluation, where fine-tuning on the target dataset is allowed. Indeed, tracking-by-detection approaches are generally more robust than those based on end-to-end learning, so much so that it is an established practice of researchers to present the results in separate parts of a table [10, 43, 48], to deliver an apple-to-apple comparison.\n\nConcerning the zero-shot setting in our work, we conclude that existing tracking-by-detection trackers are more robust to domain shifts. In these approaches, the only part potentially subject to shifts is the detector (e.g., YOLOX). Instead, the motion model (e.g., Kalman Filter) and the association strategy [3, 46] are almost parameter-free procedures that are less affected by domain shifts for construction, as their design reflects strong inductive biases about human motion. For such a reason, it is our belief that the problem of domain shift in Multiple Object Tracking (MOT) should be primarily addressed in parametric approaches such as deep neural networks. For this reason, our research question focuses on query-based trackers (e.g., MOTRv2) that learn entirely from data. Our final goal is to enhance these trackers, as their end-to-end nature results can lead to challenges during domain shifts. We will include such a discussion in the final manuscript.\n\n**W2 - Detection Thresholds used in Tab 1**\n\nWe evaluated ByteTrack on MOTSynth using the default values provided in the public ByteTrack repository, specifically a minimum confidence score (min_score) of 0.1 and a track threshold (track_thresh) of 0.6. Below, we present the results for different values of these thresholds:\n\n||HOTA|IDF1|MOTA|DetA|AssA|\n|-|-|-|-|-|-|\n|_ByteTrack (min_score: 0.1, track_thresh: 0.6)_|45.7|56.4|61.8|50.1|41.9|\n|ByteTrack (min_score: 0.05, track_thresh: 0.6)|43.0|54.4|54.5|45.9|40.9|\n|ByteTrack (min_score: 0.2, track_thresh: 0.6)|45.6|56.3|61.5|49.8|41.9|\n|ByteTrack (min_score: 0.1, track_thresh: 0.2)|45.9|56.4|61.9|51.1|41.6|\n|ByteTrack (min_score: 0.1, track_thresh: 0.3)|46.0|56.5|62.1|51.1|41.7|\n|ByteTrack (min_score: 0.1, track_thresh: 0.4)|45.9|56.6|62.2|50.9|41.8|\n|ByteTrack (min_score: 0.1, track_thresh: 0.7)|45.0|55.8|60.3|48.7|41.8|\n|PASTA|53.0|57.6|62.0|56.2|50.4|\n\nThe results are close, with a slight improvement when reducing the track_thresh to 0.3 or 0.4, while min_score of 0.1 remains optimal. We will report these updated results in the final version, along with more details about the parameters used.\n\n**W3 - Choice of attributes**\n\nWe selected the attributes reported because we believe they are the most generic and applicable to various scenarios. However, users are not limited to using these specific attributes; PASTA is fully customizable to suit different needs. For instance, if users know their system will always be employed outdoors, they could omit the indoor/outdoor attributes and add a good/poor weather attribute instead. This flexibility allows PASTA to be tailored to specific requirements, enhancing its adaptability and effectiveness across various applications. This adaptability ensures that users can optimize the trakcer for their unique contexts without needing extensive retraining or adjustments.\n\nFurthermore, PASTA's modular nature means that users can easily integrate additional modules to handle these changes as new attributes become relevant or new scenarios emerge. This future-proofs the system and allows it to evolve alongside technological advancements and shifting user needs. For example, attributes such as traffic density or driving side could be added in an automotive setting to improve performance. The ability to customize and extend PASTA ensures it remains a robust and versatile tool for a wide range of tracking applications.\n\n**W4 - Omitting LoRA**\n\nOmitting LoRA (or other Parameter-Efficient Fine-Tuning techniques) in PASTA would significantly impact its efficiency in scenario-specific adaptation. Without these techniques, each attribute would require a fully fine-tuned model, which poses several challenges, especially memory constraints [12,19].\nFirstly, storing a separate model for each attribute is highly storage-intensive. For instance, a PASTA module is approximately 5MB, whereas the full model exceeds 350MB. With 12 attributes, the total storage requirement for PASTA would be 410MB (350MB + 12 x 5MB). In contrast, storing 12 fully fine-tuned models would require around 4.2GB (12 x 350MB), representing a tenfold increase in storage needs.\nAdditionally, adapting an entire model to each specific condition is more time-consuming than using LoRA, as it involves optimizing a more significant number of parameters. This adaptation process must be repeated for each attribute, making it both impractical and costly. Moreover, fully fine-tuning a transformer-based architecture demands more data than a parameter-efficient approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their thorough evaluations and constructive feedback. These comments and suggestions have contributed to enhancing the overall quality of our work. Below, we summarize the strengths and weaknesses reported.
   
**Strengths:** We greatly appreciate the recognition of the **novelty** of our work by reviewers nupV, 6tD5, 4fn5, and Vx5C, as well as the **validation** of our experimental section by all reviewers. Reviewers oFC1, 6tD5, 4fn5, and Vx5C appreciated the **clarity and readability** of our writing and the effectiveness of the overview figure. Additionally, reviewers nupV, oFC1, and 4fn5 acknowledged the contribution of our work. For instance, 4fn5 remarked, *"PASTA performs excellently in handling heterogeneous scenarios, demonstrating significant innovation and research value."*

**Weaknesses:** Reviewers nupV, oFC1, and Vx5C highlighted the need for further comparisons with zero-shot or domain adaptation methods. Additionally, there was a common concern about the lack of a detailed computational analysis (4fn5, Vx5C). To address these weaknesses, we compared our methods with related zero-shot and domain adaptation MOT methods (oFC1) and evaluated other by-detection methods in a zero-shot setting (Vx5C). We also elaborated on the choice of attributes (nupV, 6tD5, 4fn5, Vx5C).
- nupV major concern was related to a missing experiment and ablations and required additional clarity on the attribute availability. We report the real-to-real zero-shot evaluation on PersonPath22 and further ablations as requested. Additionally, we clarify the doubts on the training section.
- oFC1 noted the lack of quantitative computational analysis and a study of the relationship with similar works. We report the time and memory requirements of our approach and its fine-tuned counterpart. Furthermore, we provide an analysis of related domain adaptation and zero-shot works.
- 6tD5 had minor concerns regarding the lack of experiments on MOT20 and the performance on “strong” attributes. We provide a motivation for this lack and the attribute choice.
- 4fn5 suggested several clarity improvements, which we will integrate in the final revision of the paper. Their major concern is related to the choice of the attributes and the computational analysis. We provide a discussion on the attribute selection and a quantitative computational analysis.
- Vx5C highlighted the need for a zero-shot evaluation of ByteTrack and other tracking-by-detection methods. We report this comparison on PersonPath22 and a threshold analysis on MOTSynth. We also provide an explanation of the attribute choice and an analysis of performance without LoRA.

We address each reviewer's concerns and recommendations in detail within our individual responses to each review. We will ensure that these improvements will be incorporated into the final revision of our manuscript.

Meta-review: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with three borderline accepts, one weak accept and one accept), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.
Decision: Accept (poster)