Is Multiple Object Tracking a Matter of Specialization?

Authors: gianluca  mancusi
mattia  bernardi
aniello  panariello
angelo  porrello
rita  cucchiara
simone  calderara
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:02:57

Abstract: End-to-end transformer-based trackers have achieved remarkable performance on most human-related datasets. However, training these trackers in heterogeneous scenarios poses significant challenges, including negative interference - where the model learns conflicting scene-specific parameters - and limited domain generalization, which often necessitates expensive fine-tuning to adapt the models to new domains. In response to these challenges, we introduce Parameter-efficient Scenario-specific Tracking Architecture (PASTA), a novel framework that combines Parameter-Efficient Fine-Tuning (PEFT) and Modular Deep Learning (MDL). Specifically, we define key scenario attributes (e.g, camera-viewpoint, lighting condition) and train specialized PEFT modules for each attribute. These expert modules are combined in parameter space, enabling systematic generalization to new domains without increasing inference time. Extensive experiments on MOTSynth, along with zero-shot evaluations on MOT17 and PersonPath22 demonstrate that a neural tracker built from carefully selected modules surpasses its monolithic counterpart. We release models and code.

Main paper: 
 introduction: 
 Video Surveillance is essential for enhancing security, supporting law enforcement, improving safety, and increasing operational efficiency across various sectors. In this respect, Multiple Object Tracking (MOT) is a widely studied topic due to its inherent complexity. Nowadays, MOT is commonly tackled with two main paradigms: tracking-by-detection (TbD) [3, 51, 61, 28, 43, 38, 29] or query-based tracking [56, 58, 63, 12] (i.e., tracking-by-attention). Although tracking-by-detection methods have proven effective across multiple datasets, their performance struggles to scale on larger datasets due to the non-differentiable mechanism used for linking new detections to existing tracks. To this end, query-based methods are being employed to unify the detection and association phase.Nevertheless, training such end-to-end transformer-based methods presents significant challenges, as they tend to overfit specific scenario settings [37, 55] (e.g., camera viewpoint, indoor vs. outdoor environments), require vast amounts of data [63] , and incur substantial computational costs. Moreover, these methods degrade under domain shifts, struggling to outperform traditional TbD methods.In light of these challenges, we propose a novel framework, Parameter-efficient Scenario-specific Tracking Architecture (PASTA), aimed at reducing the computational costs and enhancing the transfer capabilities of such models. Leveraging Parameter Efficient Fine-Tuning (PEFT) techniques [16, 34] can significantly decrease computational expenses and training time, starting with a frozen backbone pre-trained on synthetic data. However, the model may still experience negative interference [49, Composed modulesFigure 1 : Given a scene, we select the modules corresponding to its attributes, such as lighting and indoor/outdoor. These modules are composed and then deployed, yielding a specialized model. 50, 37, 55] , a phenomenon for which training on multiple tasks (or scenarios) causes the model to learn task-specific parameters that may conflict. For instance, if the model learns parameters tailored for an indoor sports activity, it could detrimentally affect its performance on a novel outdoor scene depicting people walking. To this end, inspired by Modular Deep Learning (MDL) [35] , we employ a lightweight expert module for each attribute, learn them separately, and finally compose them efficiently [17] . This approach -depicted in Fig. 1 -is akin to a chef preparing a pasta dish. Each ingredient (i.e., module) is prepared individually to preserve its unique flavor and then combined harmoniously to create a balanced dish. Moreover, as pasta must be perfectly al dente to serve as the ideal base for various sauces, the pre-trained backbone should be robust and well-tuned to serve as the foundation for the modules. These modules must be combined effectively to ensure the model performs well across diverse scenarios. Conversely, the result will be sub-optimal if incompatible modules are mixed -analogous to combining ingredients that do not complement each other. Indeed, combining contrasting modules can lead to ineffective handling of diverse tasks.Notably, such a modular framework brings two advantages: it avoids negative interference and enhances generalization by leveraging domain-specific knowledge. Firstly, starting from a pre-trained backbone, we train each module independently to prevent parameter conflicts, ensuring that gradient updates are confined to the relevant module for the specific scenario. This assures that parameters learned for one attribute do not negatively impact the performance of another. Secondly, the modular approach allows us to exploit domain knowledge fully, even when encountering a novel attribute combination. Indeed, as shown in Sec. 5.5, our approach is effective even in a zero-shot setting (i.e., without further fine-tuning on the target dataset). Moreover, the selection of the modules may be done automatically or in a more realistic production environment by video surveillance operators.To evaluate our approach, we conduct extensive experiments on the synthetic MOTSynth [10] and the real-world MOT17 [8] and PersonPath22 [44] datasets. The results show that PASTA can effectively leverage the knowledge learned by the modules to improve tracking performance on both the source dataset and in zero-shot scenarios. To summarize, we highlight the following main contributions:• We propose PASTA, a novel framework for Multiple Object Tracking built on Modular Deep Learning, enabling the fine-tuning of query-based trackers with PEFT techniques. • By incorporating expert modules, we improve domain transfer and prevent negative interference while fine-tuning MOT models. • Comprehensive evaluation confirms the validity of our approach and its effectiveness in zero-shot tracking scenarios.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The abstract and introduction sections clearly articulate the main contributions and scope of the paper, providing an accurate overview of its objectives and the significance of its findings.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Multiple Object Tracking. The most widely adopted paradigm for Multiple Object Tracking (MOT) is tracking-by-detection (TbD) [3, 51, 61, 28, 43, 38] . First, an object detector (e.g., YOLOX [14] ) localizes objects in the current frame. Next, the association step matches detections to tracks from the previous frame by solving a minimum-cost bipartite matching problem, with the association cost defined in various forms (e.g., IoU [3, 61] , GIoU [40] , or geometrical cues [29, 33] ). This pairing typically occurs immediately after propagating the previous tracks to the current frame using a motion model (e.g., Kalman Filter [19] ). Notably, methods following such paradigm have succeeded on complex human-related MOT benchmarks [8, 9, 47, 44] . In TbD, the detection and data-association steps are equally crucial to accurately localizing and tracking objects. Recent works [65, 1] have attempted to unify these steps; however, progress toward a fully unified algorithm was constrained by a significant limitation -the data association process (e.g., the Hungarian algorithm [20] ) is inherently non-differentiable. An initial effort was made by Xu et al. [53] that proposed a differentiable version of the Hungarian algorithm, later advanced by end-to-end transformer-based trackers [31, 58, 63, 56, 13] .However, transformer-based trackers (also known as tracking-by-attention) require large amounts of data to achieve decent generalization capabilities [58, 31] . Due to the data scarcity in MOT, these models often overfit to the specific domain they were trained on, which hampers their ability to generalize to different domains [17, 21, 37] .Modular Deep Learning (MDL). Considering recent trends in the field of deep learning, stateof-the-art models have become increasingly larger. Consequentially, fine-tuning these models has become expensive; concurrently, they still struggle with tasks like symbolic reasoning and temporal understanding [35] . Recent learning paradigms based on Modular Deep Learning (MDL) [35] can address these challenges by disentangling core pre-training knowledge from domain-specific capabilities. By applying modularity principles, deep models can be easily edited, allowing for the seamless integration of new capabilities and the selective removal of existing ones [26, 36] .Specifically, lightweight computation functions named modules are employed to adapt a pre-trained neural network. To do so, several fine-tuning techniques could be used to realize these modules, such as LoRA [16] , (IA) 3 [25] , and SSF [23] . These multiple modules can be learned on different tasks such that they can specialize in different concepts [32] . At inference time, not all modules have to be active at the same time. Instead, they can be selectively utilized as needed, either based on prior knowledge of the domain or dynamically in response to the current input. To establish which modules to activate, it is common practice to rely on a routing function, which can be either learned or fixed. Finally, the outputs of the selected modules are combined using an aggregation function. To minimize inference costs, this process is usually performed in the parameter space rather than the output space, an activity often referred to as model merging [54] . Specifically, a single forward pass is performed using weights generated by a linear combination of those selected by the routing function.Currently, domain adaptation techniques have only been applied to tracking-by-detection methods, with GHOST [43] and DARTH [42] serving as notable examples. In particular, GHOST adapts the visual encoder employed to feed the appearance model by updating the sufficient statistics of the Batch Normalization layers during inference. In contrast, our approach regards tracking-by-attention approaches and adapts the entire network. Moreover, DARTH employs test-time adaptation (TTA) [24] and Knowledge Distillation, requiring multiple forward passes and entire sequences, making it computationally heavy and less practical for real-time use. In contrast, our method is entirely online and requires only basic target scene attributes, with no further training during deployment.Recent advances in zero-shot tracking have focused on open-vocabulary tracking, where the model can track novel object categories by prompting it with the corresponding textual representation. In this respect, methods like OVTrack [22] and Z-GMOT [48] leverage CLIP [39] and language-based pre-training, while OVTracktor [7] extends tracking to any category. Our method does not use open-vocabulary models but emphasizes domain knowledge transfer in end-to-end trackers.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Appendix G discusses the limitation of relying on expert knowledge, specifically using a static router for the attributes. This approach necessitates annotating each attribute within the dataset, which can be time-consuming.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Efficient fine-tuning. Given the substantial size of recent vision backbones, often consisting of hundreds of millions of parameters, adapting them to new scenarios is computationally expensive, both in terms of time and memory requirements. To tackle the above problems, Parameter Efficient Fine-Tuning (PEFT) started to take place in recent literature. Among these methods, Low-Rank Adaptation (LoRA) [16] excels at such purpose. Specifically, LoRA adapts a pre-trained weight matrix θ 0 ∈ R d×k , with d and k being the dimensions of the matrix, by leveraging a low-rank decomposition θ 0 + ∆θ = θ 0 + BA, where B ∈ R d×r , A ∈ R r×k , and r ≪ min(d, k). During training, θ 0 is kept frozen, while the smaller A and B matrices are instead trainable, making the process highly efficient. The forward pass becomes h = θ 0 x + BAx, where x are the input features. Deformable A.Q K V Deformable A. Q K V Self Attention Q K VFigure 2 : Overview of our modular architecture. A domain expert selects PEFT modules based on sequence attributes such as lighting and camera movement. These selected modules are then composed and applied to each model layer, adapting the backbone and encoder-decoder architecture.Query-based Multiple Object Tracking. The underlying backbone of our transformer-based tracker follows the structure of [58] . In a nutshell, such a query-based model forces each query to recall the same instance across different frames. Specifically, we leverage an end-to-end trainable tracker built upon the Deformable DETR [6] framework conditioned by the image features extracted with a convolutional backbone (i.e., ResNet [15] ). Following [63] , we further condition the DETR decoder with a set of detections from an external detector network and a shared learnable query.At time t = 0, new proposals are generated from the objects detected in the scene. These proposals are then updated through self-attention and interact with image features via the deformable attention layer. The final prediction output is the summation of the initial anchors and the predicted offsets.For subsequent frames (t > 0), track queries generated from the previous frame are concatenated with learnable proposal queries of the current frame. Moreover, previous predictions are integrated with current proposals to establish new anchors for the incoming frame. We refer the reader to the original paper [63] for further details. It is noted that the flexibility of this architecture allows for the seamless integration of techniques based on modularity.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
method: 
 We herein present PASTA, depicted in Fig. 2 , a novel approach to Multiple Object Tracking that leverages PEFT modules to enable attribute-specific module specialization and reuse. This approach allows for the dynamic configuration of an end-to-end tracker by selecting the appropriate modules for each input scene, fully leveraging heterogeneous pre-training while avoiding negative transfer.Attribute-based modularity. We devise a set of learnable modules to fine-tune each layer of our query-based tracker. Each module is related to an attribute: as shown in Fig. 3 , we define N = 5 attributes, namely lighting, viewpoint, occupancy, location, and camera motion, and provide a tailored module for each discrete value these attributes take (see Sec. 5.3 for details). For instance, the location attribute has indoor and outdoor modules. At inference time, prior knowledge about the input scene is used to determine the appropriate value for each attribute, which in turn selects the corresponding modules from the "inventory", denoted as M .Since the base model [63] During training, we start with the pre-trained weights and integrate all the modules while keeping the original parameters frozen. To prevent negative interference, we optimize each module independently, randomly sampling one attribute at a time and updating only the corresponding module at each training iteration. By the end of the training process, we obtain a set of specialized parameters (experts), which can be seamlessly merged during inference to improve overall tracking performance.Routing through Domain Expert. During inference, two essential steps are required to exploit the learned modules: routing and aggregation. With multiple modules available from the inventory M , a routing strategy is required to determine the modules that should be active. To make this selection, we draw on what is known in the literature as expert knowledge [52, 35] (or "Domain Expert" in Fig. 2 ). In real-world applications such as video analytics, the expertise guiding the selection can come from a video surveillance operator or human analyst, who configures the appropriate modules to reflect domain-and scene-specific settings, such as camera perspective, lighting conditions, and other critical details. This approach allows users to optimize the tracking module for their unique contexts without extensive retraining. Additionally, the modular nature of the system enables easy integration of new modules to address emerging attributes or scenarios.Relying on Domain Expert to select attributes is a grounded practice in real-world applications. For instance, the camera's mounting perspective and whether the scene is indoors or outdoors are typically known factors in fixed-camera scenarios. Additionally, automatic approaches can be envisioned to minimize human intervention further. For example, lighting conditions can be inferred by analyzing brightness levels, and a detector can count objects of interest in the scene, classifying crowd density.Modules composition. In the final step, we aggregate the selected modules ("Modules Composition" in Fig. 2 ) and incorporate the result into the pre-trained tracker to create an expert model. Since these modules have been obtained by fine-tuning from θ 0 , each module θ ⋆ corresponds to a specific displacement τ ⋆ = θ ⋆ -θ 0 in parameter space relative to the initial pre-training parameters θ 0 . This displacement is known as the task vector [18] . The final composed model f (•; θ c ) is defined as:EQUATIONWhen λ i = 1 N , the formula simplifies to the average of the task vectors corresponding to each attribute. We employ this straightforward strategy for λ i , giving equal weight to all attributes. However, considering the task vector τ i associated with the i-th attribute, we employ a more sophisticated approach. If there are no domain shifts during inference (i.e., both training and testing occur on the same dataset, such as MOTSynth), the task vector τ i is simply set to the displacement τ ⋆ produced by the expert module selected by the Domain Expert. In contrast, when domain shifts are present (e.g., training on MOTSynth and testing on MOT17), we adopt a soft strategy that considers all the modules in the inventory associated with the relevant attribute. In doing so, we follow the insights from [59] , where the authors demonstrated that scenarios with shifting tasks benefit from richer representations than those derived from a single optimization episode. Specifically, given the i-th attribute, let R(i) be the set of its modules. We recall that each attribute admits multiple discrete values (e.g., R(occupancy) = {"low", "medium", "high"}), and different attributes may have different cardinalities (e.g., |R(occupancy)|= 3 and |R(lighting)|= 2, as detailed in Sec. 5.3). Building on this, we employ soft routing to create the corresponding task vector, assigning the largest portion of the cake, e.g. ρ = 0.80, to the module selected by the Domain Expert. The remaining modules are weighted by (1 -ρ)/(|R(i)|-1), ensuring that the total sum equals 1. For example, considering those layers fine-tuned with the LoRA, the corresponding task vector is computed as:τ i = m∈R(i) λm B m A m , where λm = ρ if m is selected, 1-ρ |R(i)|-1otherwise.(EQUATIONNote that when ρ = 1, the soft strategy becomes hard, meaning that only the module selected by Domain Expert is utilized. By applying the formula above to all attributes, we obtain N task vectors, which we aggregate following Eq. ( 1).Similarly, we apply channel-wise scale and shift [23] operations to adapt each backbone layer. Formally, given the output F of a convolutional layer, the i-th module applies a scale & shift operation to obtain the edited Fi , such that Fi = γ i ⊙ F + β i with ⊙ denoting the Hadamard product.At inference time, we combine the output of different scale & shift modules by noting thatEQUATIONwhich means that parametrizing the scale & shift layer with a simple weighted average effectively results in averaging the outputs of the corresponding individual layers. The formula above applies to the in-domain setting but can be easily generalized to the soft routing scheme outlined by Eq. ( 2). Eventually, as discussed in [23] , the scale & shift layer can be absorbed into the previous projection layer, thus ensuring that the inference process incurs no additional computational costs. The same re-parametrization trick can be employed to extract the task vector underlying scale & shift fine-tuning (refer to appendix A for additional notes). 
datasets: 
 MOTSynth [10] is a large synthetic dataset for pedestrian detection and tracking in urban scenarios, generated using a photorealistic video game. It comprises 764 full HD videos, each 1800 frames long, showcasing various attributes. In our experiments, following [29] , we reduced the test sequences to 600 frames each and further split the training set to extract 48 validation sequences, shortened to 150 frames, for validation during training.PersonPath22 [44] is a large-scale pedestrian dataset consisting of 236 real-world videos featuring longer occlusions and more crowded scenes. It is divided into 138 training videos and 98 test videos.MOT17 [8] is a well-known benchmark, containing 7 sequences for training and 7 for testing, with different image resolutions, featuring crowded street scenarios with both static and moving cameras.We evaluate our proposed PASTA on both in-domain and out-of-domain scenarios. For the indomain evaluation, we train and test PASTA on the MOTSynth synthetic dataset (Sec. 5.4) using expert modules in a domain-specific context. As a baseline, we train [63] on MOTSynth without using modules, referring to this model as MOTRv2-MS. For the out-of-domain evaluation, we conduct a synth-to-real zero-shot experiment on MOT17 and PersonPath22 (Sec. 5.5). Starting from training on MOTSynth, we test PASTA on these datasets without additional training, showcasing its ability to generalize under non-identically distributed domains. Finally, we present a series of ablation studies in Sec. 6 to take a closer look at the effectiveness of our method.Competing trackers and metrics. In addition, we report the performance of other notable methods, including strong tracking-by-detection baselines such as ByteTrack [61] and OC-Sort [5] . We also include evaluations of query-based trackers, such as TrackFormer [31] and MOTRv2 [63] (see MOTRv2-MS). To compare their performance, we employ five metrics, ordered from detection to association, as recommended by [42] . These metrics are DetA [27] , MOTA [2] , HOTA [27] , IDF1 [41] , and AssA [27] . For the PersonPath22 dataset, we use their official metrics, MOTA and IDF1, supplemented by FP (false positives), FN (false negatives), and IDSW (identity switches). We initialize our models using the pre-trained weights from DanceTrack [47] , as provided by the authors of [63] . We employ YOLOX [14] as the auxiliary detector, exploiting weights from ByteTrack [61] . To provide a shared initialization for both PASTA and MOTRv2-MS training, we train a bootstrap model starting from the DanceTrack pre-train for 28k iterations on the MOTSynth training set. This bootstrap initialization uses half of the original training sequences from MOTSynth to align our model with the scenarios represented in the dataset. The learning rates are set to 5 × 10 -5 for the transformer and 1 × 10 -6 for the visual backbone.In the second phase, we deploy the PEFT modules to fine-tune the bootstrap model. By excluding half the sequences during the bootstrap, we make sure that the modules can still learn valuable features. To ensure a fair comparison, we train each module for a similar number of iterations as MOTRv2-MS, with approximately 17k iterations. Regarding the encoder-decoder model, we apply our modularization strategy to every linear layer except those with output dimension less than 128. Attributes. We employ five key attributes to realize our modular architecture: lighting, camera viewpoint, people occupancy, location, and camera motion. For lighting, we specialize modules for good and bad lighting conditions. To do so, we threshold the brightness value V of the HSV representation at 70. The viewpoint attribute includes modules for high, medium, and low camera angles. We manually annotate this attribute as follows: i) scenes where the camera is parallel to the ground at or below pedestrian head level are labeled as "low-level"; ii) "high-level" viewpoints include vertical perspectives or scenes where the camera is positioned very high or far from people; and iii) "medium-level" includes all other camera angles. For occupancy, we design modules that reflect the crowd density within the scene: low (up to 10 people), medium (10 to 40 people), and high (more than 40 people), based on the count of detections with a confidence score above 0.2. The location attribute differentiates between indoor and outdoor settings. Lastly, the motion attribute comprises modules for both moving and static cameras, enabling the model to adapt to different camera movement scenarios. Further details on dataset statistics are provided in appendix C.To assess the impact of the negative interference, we conduct several experiments on MOTSynth (see Tab. 1). Given the wide variety of scenarios in such a synthetic dataset, one can appreciate the advantages of using specialized modules. Indeed, integrating our modules resulted in an overall improvement w.r.t. its fine-tuning counterpart (MOTRv2-MS). Specifically, we observe an improvement over the association metrics (AssA, IDF1) and the HOTA and MOTA metrics. These enhancements suggest the benefits of our approach in reducing negative interference during training. By assigning each module a specific role tailored to particular scenario settings, we achieve improved training stability through a deterministic selection process guided by a domain expert. By designing distinct modules for various input conditions, we can effectively select the appropriate modules to handle distribution shifts, such as transitions to a new domain. We assess the benefits of this ability using synthetic data for training, and then evaluate on new, unseen datasets without any additional re-training (zero-shot). To do this, we start with our model trained on MOTSynth as described in Sec. 5.4 and evaluate it on MOT17 (Tab. 2) and PersonPath22 (Tab. 3). While these datasets share similarities in the attributes we employed, we emphasize that the source dataset is synthetic and the targets are real-world, resulting in a significant shift.The results reported in Tab. 2 and 3 show an improvement over the baseline (i.e., MOTRv2-MS), with +1.4 in HOTA and +1.9 in IDF1 in zero-shot MOT17, and +1.7 in MOTA and +0.7 in IDF1 in PersonPath22. Our approach demonstrates better generalization capabilities, helping close the gap with fully-trained methods while less computationally demanding. These results indicate that modularity enhances performance within the source dataset and improves domain generalization, leading to more reliable and versatile approach for tracking. Furthermore, other than reporting the results with the standard module selection (considering only the modules present in the scenes, ρ = 1), we also experiment with the weighted aggregation of all modules (ρ = 0.8) (detailed in Sec. 4). Interestingly, while the standard strategy shows improvements, the weighted aggregation strategy yields better performance. This suggests that richer representations, obtained by including multiple modules per attribute, are more effective for zero-shot scenarios than a single-module approach [59] . Evaluating zero-shot real-to-real transfer. In Tab. 4, we present an additional experiment to evaluate the performance of PASTA in a zero-shot setting, this time using a realistic dataset as the source, rather than a synthetic one. For comparison, we train MOTRv2 on the MOT17 dataset and assess its performance on PersonPath22. Our approach showcases superior results compared to the fine-tuned MOTRv2, highlighting that leveraging modules enhances the model, with improved generalization capabilities in new and real-world domains. 
ablation studies: 
 In Tab. 5, we evaluate the effect of various routing and aggregation strategies in both the in-domain setting (MOTSynth, left side of Tab. 5) and the zero-shot setting (MOT17, right side of Tab. 5). In the in-domain scenario, the results show that averaging the modules selected by the Domain Expert, specifically using Mean avg. (ρ = 1.0), is the most effective strategy. We also experimented with summation, as proposed by [60] , but this method produced bad results, which we impute to the alteration of weight magnitudes when summing multiple modules. Another noteworthy approach is the weighted avg., described in Sec. 4, which incorporates all modules, including those not selected.While using only the selected modules is the optimal strategy in the in-domain scenario, for the zero-shot case (MOT17), incorporating knowledge from the non-selected modules -specifically, using Weighted avg. (ρ = 0.8) -enhances tracking performance. This pattern is also consistent when the domain shift involves evaluation on the PersonPath22 dataset (see appendix E).Module selection. Should we select only the modules representing the current scenario, as determined by the Domain Expert approach, or would performance improve by incorporating all available modules? In Tab. 5, we investigate this matter by comparing these two approaches. To provide a more comprehensive perspective, we also evaluate a strategy that, in stark contrast to the Domain Expert, selects the opposite modules (e.g., selecting the outdoor and poor lighting modules when presented with an indoor, well-lit scene). The lowest performance is observed when using opposite modules, indicating that using the proper modules provides valuable information about the current scene. Interestingly, the model still performs relatively well despite using opposite attributes, likely due to contributions from other modules whose general knowledge of the domain sustains overall performance. This suggests that modules can assist one another in solving tasks. negative interference -achieved by training each module separately -prevents the modules from relying on each other and allows them to make unique contributions independently.Furthermore, in Fig. 4 , we illustrate how the incremental addition of specialized modules improves IDF1 and HOTA metrics, showcasing that greater specialization of the modules gradually enhances overall performance. For a more detailed analysis, in Tab. 6, we select the opposite modules instead of the correct one for each attribute. Although the metrics are further reduced, the model performs well due to its robust pre-training, as indicated by the no modules baseline shown in the table.Block-wise analysis. In our approach, attribute-related modules are applied to edit the entire network. However, users may opt to edit selectively specific parts of the architecture, thereby identifying which components are most critical. In Tab. 7, we conduct an ablation study by excluding our modules from being applied to varying components of the architecture. The results indicate that not applying task vectors to the decoder significantly degrades detection and association metrics. We believe that this degradation can be explained by considering the crucial role of the decoder. The decoder must indeed gather information from detection, tracking, and proposal queries while simultaneously integrating visual information from the encoder. Consequently, not adapting the decoder prevents the architecture from effectively leveraging queries and visual cues. The encoder also contributes substantially, though to a lesser extent than the decoder, as it primarily refines and contextualizes visual features from the backbone. Finally, the backbone shows the smallest contribution. 
conclusions: 
 In this work, we introduce PASTA, a novel framework that enhances domain generalization in tracking-by-query methods for Multiple Object Tracking. Our approach features a modular structure with dedicated modules tailored to different attributes of real-world scenes. These modules utilize Parameter-Efficient Fine-Tuning techniques, enabling the integration of scene-specific parameters while minimizing computational load. Comprehensive experiments demonstrate that domainspecialized modules significantly bolster robustness, allowing effective adaptation across domains without extensive retraining. PASTA further enables camera operators to configure the optimal module for each unique scenario, ensuring precise adaptation to diverse real-world settings. outperforms the standard one across all metrics, demonstrating the effectiveness of the modular training in mitigating catastrophic forgetting. Indeed, the LoRA [16] modules act as a regularizer that mitigates forgetting of the source-domain [4] . 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Societal impacts are reported in appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the original authors have been properly cited, and the relevant licenses are reported in the appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To further support our claim on leveraging all modules in a zero-shot scenario, we conduct an additional ablation study on the test split of PersonPath22. As shown in Tab. C, this test confirms that retaining knowledge from modules not directly related to the specific scenario is beneficial when dealing with domain shifts. Specifically, our selection strategy outperforms the unweighted average by empirically assigning a weight ρ = 0.8 to the selected modules and ρ = 0.2 to the others.Comparison with tracking-by-detection. To comprehensively evaluate our method, we herein test ByteTrack and other tracking-by-detection methods in a zero-shot setting from MOT17 to PersonPath22. We report the results on PersonPath22 in Tab. D. Results indicate that PASTA leads to remarkable improvements compared to the other query-based end-to-end approach (i.e., MOTRv2 [63] ), even though they are both outperformed by the tracking-by-detection methods (such as ByteTrack [61] ). To be more comprehensive, PASTA remains competitive in terms of association performance (IDF1), but it yields weaker detection capabilities. Such a trend does not surprise us and is in line with what occurs in the more standard evaluation, where fine-tuning on the target dataset is allowed. Indeed, tracking-by-detection approaches are generally more robust than those based on end-to-end learning, so much so that it is an established practice to present the results in separate parts of a table [12, 58, 63] , to deliver an apple-to-apple comparison.In a zero-shot setting, we conclude that existing tracking-by-detection trackers are more robust to domain shifts. In these approaches, the only component potentially subject to shifts is the detector (e.g., YOLOX [14] ). Instead, the motion model (e.g., Kalman Filter [19] ) and the association strategy [3, 61] are almost parameter-free procedures that are less affected by domain shifts for construction, as their design reflects strong inductive biases about human motion. For such a reason, it is our belief that the problem of domain shift in Multiple Object Tracking (MOT) should be primarily addressed in parametric approaches such as deep neural networks. For this reason, our research question focuses on query-based trackers (e.g., MOTRv2) that learn entirely from data. Our final goal is to enhance these trackers, as their end-to-end nature results can lead to challenges during domain shifts.ByteTrack thresholds. In Tab. 1, we evaluated ByteTrack on MOTSynth using the default inputparameters provided in the public ByteTrack repository, specifically a minimum confidence score (min_score) of 0.1 and a track threshold (track_thresh) of 0.6. In Tab. E, we present the results for different values of these thresholds. The results are close, with a slight improvement when reducing the track_thresh to 0.3 or 0.4, while min_score of 0.1 remains optimal.F On computational costs Memory efficiency. We compare the GPU memory of full fine-tuning versus our approach on the MOTSynth dataset. Our method reduces training GPU memory requirements from 13GB to 8.25GB (for a batch size of 1), a reduction of over 35%. This significant decrease is due to the lower number of parameters updated by the optimizer: 42M parameters for standard fine-tuning versus 15M for our PEFT technique, as reported in Tab. 1.Inference speed. Additionally, our approach does not add any overhead during inference, aside from weight merging, which is negligible for stationary attributes, compared to MOTRv2, which maintains a speed of 6.9 FPS on a 2080Ti GPU.Storage efficiency. Using PEFT techniques significantly reduces storage needs. Without these techniques, each attribute would require a fully fine-tuned model, which poses several challenges, especially in memory constraints [16, 25] . Firstly, storing a separate model for each attribute is highly storage-intensive. For instance, a PASTA module is approximately 5MB, whereas the full model exceeds 350MB. With 12 attributes, the total storage requirement for PASTA would be 410MB (350MB + 12 x 5MB). In contrast, storing 12 fully fine-tuned models would require around 4.2GB (12 x 350MB), representing a tenfold increase in storage needs. Additionally, adapting an entire model to each specific condition is more time-consuming than using LoRA, as it involves optimizing a more significant number of parameters. This adaptation process must be repeated for each attribute, making it both impractical and costly. Moreover, fully fine-tuning a transformer-based architecture demands more data than a parameter-efficient approach.One limitation of our approach is the reliance on an expert router, which requires manual data annotation or intervention by an external domain expert. This process can be resource-intensive and may not scale well for larger datasets or diverse scenarios. Future work may explore the development of automatic routing techniques, which could significantly improve scalability, performance, and ease of deployment in real-world applications by reducing the dependency on manual annotations.Positive Impacts. Enhanced security and surveillance is one of the key benefits of this work. Improved accuracy and robustness in tracking can lead to better crime prevention, more efficient law enforcement, and increased public safety. Additionally, operational efficiency is another positive impact, where various sectors, including transportation, retail, and urban planning, can benefit from optimized operations and resource allocation. Moreover, customization and adaptability are enhanced by tailoring modules for specific scenarios, increasing versatility in applications ranging from healthcare to sports analytics.Negative Impacts. However, there are also potential negative impacts to consider. Privacy concerns arise from increased tracking capabilities, which may lead to unauthorized surveillance and privacy infringement. Bias and fairness are also issues, as biased training data can perpetuate existing biases, leading to unfair treatment of certain groups.While the modular approach presents significant advancements, it is crucial to address these societal impacts through careful design and transparent policies.NeurIPS Paper Checklist 


Decision: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with three borderline accepts, one weak accept and one accept), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.

Reviews: review: paper_summary: paper_summary: The paper presents a multiple object tracking framework that can generalize to new domains by training specialized modules for each scenario attributes.
These modules are trained using Parameter-Efficient Fine-tuning and modular deep learning techniques on a transformer-based tracker. 
This tracker is build on the Deformable DETR framework with a ResNet backbone for image feature extraction.
Extensive experiments are performed on both synthetic and real datasets to compare with some recent track-by-detection and track-by-attention approaches.
It shows that the proposed approach can generalize well on unseen datasets and reduce negative interference during training.
strength_weakness: strength_weakness: 1. The authors only conduct a synth-to-real zero-shot experiment. It would be better to see real-to-real experiment where one can train on one dataset and test on a new dataset.
2. It would be great to have some statistic on the training dataset in terms of the attributes, e.g. how many frames / videos have high/low occupancy, etc. 
3. One missing aspect in the ablation studies is how the task vector is incorporate to the pre-trained tracker contribute to the final results. For examples, incorporating on the backbone, encoder or decoder only and the whole network can have different results.
4. The proposed approach seems to be limited by the availibility of the attribute data for training each scenario-specific module.
questions: questions: 1. Is it necessary to have training data for all possible cases to train all the parameter modules?
2. How does the model trained on the MOT17 training set perform on the PersonPath22 test set? This will also demonstrate the generalization ability of the model in the zero-shot setting on a new domain.
3. It is not clear why MOTRv2 is not compared in the fully-trained section in Tables 2 and 3 while TrackFormer [24] is included in both fully-trained and zero-shot.
limitations: limitations: Yes. The authors provided some analysis on limitations and social impacts.
paper_summary: paper_summary: The paper introduces a fine-tuning framework, denoted PASTA, for multiple-object tracking that is aimed at reducing the cost of tine-tuning large models while mitigating negative inference to improve zero-shot transfer and domain generalization.

During training, the authors independently fine-tune per-domain modules/experts on pre-defined scenario attributes (e.g. lighting, static or moving camera, ...). 
Modules only adapt a sub-set of the hyperparameters and the adaptation is done by relying on low-rank adaptation (LoRA). This allows for fine-tuning with a low computational and memory footprint.
More specifically, the authors only adapt batch norm parameters in the backbone and linear layers in the transformer modules.

During inference, the method relies on "expert knowledge" to select the right module for each attribute (e.g. select the "bad lighting" module for the "lighting" attribute). A convex combination of the attribute's modules' weights is then applied (with the selected module having a higher coefficient, defined by a hyper-parameter).

For the zero-shot (transfer) setting, the model is simply set to be a weighted average of all modules.

Experiments demonstrate the effectiveness of the method by comparing performance with the same model trained without the framework. PASTA outperforms the baseline on both the synthetic dataset and on zero-shot transfer (from synthetic to real-world datasets). The authors further demonstrate the reduced forgetting in the ablation studies.
strength_weakness: strength_weakness: - The approach is claimed to reduce fine-tuning costs (reduced time and memory footprint), but no time or memory comparison to full tine-tuning is provided.
- The approach is related to domain adaptation and zero-shot transfer, but does not discuss similarities and differences to those approaches in the related work section, and does not compare to such methods in the experimental section.
questions: questions: - What is the computational gain in terms of time and memory in comparison to the full fine-tuning of MOTRv2?
- How does the method relate to domain adaptation and zero-shot methods?
limitations: limitations: Yes, the authors discuss the potential positive and negative societal impacts of multiple-object tracking, as well as the main limitation of the method (i.e. the reliance on an external "domain expert")
paper_summary: paper_summary: This paper aims to address the domain gap across different multi-object tracking datasets. The proposed method is inspired by LoRA and introduces parameter-efficient fine-tuning for state-of-art end-to-end trackers. Specifically, these trackers are first trained on a large-scale dataset MOTSynth. After that, by shifting several expert attributes and training partial parameters, these trackers can achieve impressive performance on MOT17 and PersonPath22.
strength_weakness: strength_weakness: The idea of this paper is attractive for me. However, I still have several minor concerns about the experiments:
1.This paper does not perform zero-shot learning on another popularly used dataset MOT20.
2.The employed attributes in this work, such as lighting, viewpoint, occupancy, location, and camera, seem “weak”. How about the transferring performance on “strong” attributes, like scenes and categories? In other words, how about the zero-shot evaluation performance on other datasets like DanceTrack and KITTI?
3.In Table 1, both model parameters and trainable parameters should be listed for a better comparison. Besides, the table captions in experiments are suggested to provide more descriptions.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Please refer to the weaknesses.
paper_summary: paper_summary: This paper proposes a new framework called PASTA (Parameter-Efficient Scenario-specific Tracking Architecture), which aims to improve the generalization ability of multi-object tracking (MOT) in diverse scenarios. The main contributions of this paper include: 1) proposing the PASTA framework to achieve efficient query tracker fine-tuning through PEFT technology; 2) improving domain transfer and preventing negative interference by introducing expert modules; 3) validating the method's effectiveness in zero-shot tracking scenarios through a comprehensive evaluation.
strength_weakness: strength_weakness: 1，The PASTA framework relies on manual selection by domain experts to determine the appropriate modules for the current scenario. Although effective, this method has limitations in practical applications. In some scenarios, it may not always be possible to obtain support from domain experts, or the judgments of domain experts may be influenced by subjective factors, affecting the overall performance of the model. Additionally, this manual selection method may lead to consistency and repeatability issues and is challenging to automate in large-scale applications.
2，Although the PASTA framework has been extensively tested on the MOTSynth, MOT17, and PersonPath22 datasets, these experiments mainly focus on pedestrian tracking and surveillance scenarios. There is a lack of validation in other important application fields (such as traffic monitoring, smart retail, autonomous driving, etc.), limiting the generality and promotion of the results. Further experiments can help verify the application effects of PASTA in broader scenarios, enhancing its generality and practical value.
3，The paper mentions that PASTA reduces computational costs through parameter-efficient fine-tuning, but it lacks detailed analysis of specific computational resources and time costs. For example, in practical applications, how much computational resources (such as GPU/CPU time, memory requirements, etc.) are specifically needed, and how much time is saved compared to traditional methods. Detailed analysis of resources and time costs can help evaluate the practical feasibility of PASTA in different application scenarios.
4，Although the paper mentions the advantages of modular design, it lacks an in-depth study of the potential synergy effects between different module combinations. There may be complementary or conflicting effects between different modules, which have significant impacts on the final performance. Conducting relevant experiments and analysis can help understand which module combinations are most effective, thereby optimizing the design and application strategies of PASTA, further enhancing model performance.
5，The paper does not provide a detailed discussion on the robustness of model parameters in the PASTA framework. There is a lack of experiments testing the model's performance under different noise levels, data quality, and data volumes. These tests can verify the model's stability and robustness in practical applications, further improving the credibility and practicality of PASTA. These experiments can demonstrate the adaptability and stability of PASTA in various real-world environments, ensuring it maintains high performance under various conditions.
questions: questions: 1，The modular design proposed in the paper effectively avoids negative interference and improves domain generalization ability. However, the selection and combination of modules can be further optimized using more intelligent methods, such as reinforcement learning or other automated techniques, instead of relying on manual selection by domain experts. Intelligent optimization of module selection can further enhance the model's adaptability in complex scenarios while reducing dependence on domain experts, thereby increasing the practical application value of the system.
2，The paper verifies the effectiveness of PASTA on multiple datasets, but it should consider validation in more real-world application scenarios, such as traffic monitoring, smart retail, autonomous driving, etc. These fields have unique tracking needs and challenges. By verifying PASTA in these scenarios, the wide applicability and practical value of PASTA can be further demonstrated, enhancing the paper's persuasiveness and showcasing PASTA's potential application effects in different fields.

3， When combining modules, the synergy between modules should be considered, and related experiments should be added to explore the performance impact of different module combinations. Specifically, experiments can be designed to evaluate which module combinations can produce the best effects or which combinations may lead to performance degradation. In-depth research on module synergy can help understand and optimize the PASTA module design, improve overall model performance, and provide valuable references for future module development.
4，Discussion of Computational Resources and Time Costs： Although PASTA reduces computational costs through parameter-efficient fine-tuning, the specific computational resources and time costs in practical applications have not been discussed in detail. Particularly in comparison with traditional methods, clearly listing the specific computational resources (such as GPU/CPU time, memory requirements, etc.) and time costs required in the training and inference stages can help readers better understand the feasibility and advantages of PASTA in practical applications. This analysis is crucial for evaluating the practical benefits of PASTA in large-scale real-world applications.
5，The discussion of future work and application prospects of the PASTA framework is relatively brief and does not fully demonstrate its long-term value and expansion potential.
limitations: limitations: 1，Complexity of Modular Systems
The modular design of the PASTA framework increases the complexity of the system, especially in terms of training, managing, and combining modules. Each module requires separate training, optimization, and validation, increasing development and maintenance costs. In practical deployment, modular systems require more complex architectural support, involving dynamic selection and combination of modules, which may increase implementation difficulty and system maintenance complexity.
2，Generalization Limitations
Although PASTA performs well on specific datasets and scenarios, its design and optimization target specific scenario attributes (such as camera angles, lighting conditions, etc.). PASTA's generalization and adaptability may be limited when dealing with new scenarios that do not belong to these predefined attributes. The framework's performance in completely different application environments or new domains has not been fully validated, potentially affecting its promotion and widespread application.
3，Dataset Dependency
The experiments in the paper are mainly based on the MOTSynth, MOT17, and PersonPath22 datasets, which may have certain biases or characteristics. The excellent performance of the PASTA framework on these datasets may not be directly generalizable to other datasets or more diverse data sources. The lack of experimental validation on more diverse datasets limits the comprehensive evaluation of PASTA's generalization ability in various scenarios.
4，Dependence on Pre-trained Models
The overall performance of the PASTA framework largely depends on pre-trained backbone networks and detectors. If pre-trained models perform poorly in certain tasks or scenarios, the performance of the PASTA framework will also be significantly affected. This means that the application and effectiveness of the PASTA framework are, to some extent, limited by the quality of pre-trained models, requiring high-quality pre-trained models to achieve optimal performance.
5，Lack of Evaluation of Real-time Performance
The paper does not provide a detailed evaluation of the real-time performance of the PASTA framework, such as processing latency and efficiency in real-time video streams. Real-time performance is critical for many practical applications (such as video surveillance and autonomous driving). The lack of evaluation in this aspect makes it difficult to judge the feasibility and effectiveness of PASTA in real-time environments, potentially limiting its application prospects in scenarios requiring high real-time performance.
paper_summary: paper_summary: The paper introduces PASTA, a framework designed to address the challenges of training end-to-end transformer-based trackers in heterogeneous scenarios, specifically negative interference and poor domain generalization. PASTA leverages Parameter-Efficient Fine-Tuning (PEFT) and Modular Deep Learning (MDL) to define and train specialized modules based on key scenario attributes like camera viewpoint and lighting conditions. These modules are then combined using task arithmetic to enhance generalization to new domains. The framework's effectiveness is demonstrated through extensive experiments on MOTSynth and zero-shot evaluations on MOT17 and PersonPath22, where it outperforms traditional monolithic trackers. The primary contributions include the introduction of PASTA, the use of PEFT and MDL for better domain adaptation, and the validation of its superior performance in diverse tracking scenarios.
strength_weakness: strength_weakness: The paper has several weaknesses that should be addressed. 
1. The authors do not compare their framework to ByteTrack and other trackers in zero-shot settings, limiting their evaluation's comprehensiveness. 
2. For a fair comparison in Table 1, the authors should provide detailed information on the different detection thresholds used, as varying thresholds can significantly affect ByteTrack's performance. 
3. The authors need to discuss their choice of attributes in more detail, explaining why specific attributes were selected and their relevance to the framework. 
4. It remains unclear if the performance could be improved without using LoRA, as the authors do not explore or report the impact of omitting LoRA in their experiments.
questions: questions: I have listed my concerns and questions above.
limitations: limitations: The authors have claimed the limitations.

Comments: comment: Thank you for your detailed and thoughtful review. As we approach the end of the rebuttal period, we wanted to check in to see if there are any further questions or concerns we can clarify. We appreciate your feedback and look forward to your response.
Thanks to the authors for addressing my concerns by providing more details about the practical gains regarding training time and memory requirements and broadening the related work. I will maintain the "accept" rating.
The authors have adequately addressed the concerns raised. I appreciate their efforts and will maintain my rating.
The authors have addressed the concerns. I keep my rating score.
Thanks for your response. The authors address my main concerns.