Enhancing vehicle-mountable multiple object tracking systems with embeddable Ising machines

Domain: physics
Published: 2025-12-17
Editorial outcome: major_revision

Abstract
The cognitive function of tracking multiple objects, essential for autonomous mobile vehicles or autonomous robots, involves object detection and their temporal associations. While significant progress has recently been made in machine learning to elaborate the similarity matrix between the objects that have been recognized and the objects detected in the current video frame, less progress has been made on the assignment problem that ultimately determines temporal associations, which is a combinatorial optimization problem. Here we show a vehicle-mountable multiple object tracking system with a flexible assignment function for tracking through multiple long-term occlusion events. To solve the flexible assignment problem, formulated as a non-deterministic polynomial-time hard problem, the system relies on an embedded Ising machine based on a quantum-inspired algorithm called simulated bifurcation. Using a vehicle-mountable computing platform, we demonstrate real-time system-wide throughput of more than 20 frames per second with the enhanced functionality.

Introduction
Autonomous control in mobile vehicles, for advanced driver assistance systems (ADAS) and toward fully automated driving, as well as in mobile robots is achieved through high-speed real-time systems that periodically execute a task pipeline comprising sensing, understanding, planning, and control1–3. Understanding the surrounding environment and egomotion involves advanced information processing, including object detection and tracking4–11, as well as localization and mapping12. These processes must be performed efficiently on vehicle-mountable computing platforms13–15, which are constrained by size, power consumption, and cost.

Multiple object tracking (MOT) is a cognitive process that identifies and maintains awareness of multiple objects despite their movement, even through temporary occlusion events such as object crossings. Modern MOT systems4–11 use a tracking-by-detection strategy, in which the objects that have been identified and continuously followed by the system (hereafter, tracks) are associated with the objects detected in the current video frame (hereafter, detections) and then updated using the information from the matched detections (see Fig. 1). The tracks thus draw the trajectories of the objects. The assignment (matching) between tracks and detections is determined based on a similarity (or association) matrix, where each matrix element corresponds to the similarity between a track and a detection. Various sophisticated definitions of similarity have recently been proposed based on advanced machine learning methodologies4–11. The assignment problem, which aims to maximize the overall likelihood and ultimately determines the temporal association of objects, is a combinatorial (or discrete) optimization problem, more specifically, a bipartite graph matching problem. The assignment problems in those MOT systems4–11 that assume one-to-one correspondence between tracks and detections are linear (i.e., they are linear assignment problems) and can therefore be solved in polynomial time using an exact algorithm known as the Hungarian algorithm16. However, during periods of object crossing (occlusion), a many-to-one correspondence (i.e., many tracks to one detection) may be more plausible. Assignment problems that consider the possibility of many-to-one correspondence are more complex combinatorial optimization problems and are difficult to solve on conventional von Neumann computers.

Following the introduction of D-wave’s quantum annealer in 201117, domain-specific computers designed to solve difficult combinatorial optimization problems in a short time17–43, known as Ising machines, have attracted attention for their potential to substantially accelerate the solution of such problems. The Ising machines aim to find the lowest-energy states of Ising spin models44, which consist of binary variables (called spins) coupled through pairwise interactions. The Ising problem, which is equivalent to quadratic unconstrained binary optimization (QUBO), belongs to the class of the non-deterministic polynomial-time hard (NP-hard) problems45,46. A wide range of computationally hard combinatorial problems can be formulated as the Ising problems46. The Ising machines have been implemented using a variety of hardware platforms18 including superconducting qubits17,26, optical systems27–29, memristor-based neural networks30, probabilistic bits31,32, spintronics systems33, coupled oscillators19,34–37, analog computing units38, application specific integrated circuits (ASICs)39,40, field programmable gate arrays (FPGAs)19–22,24,25,41,43, and graphic processing units (GPUs)19,21,42.

Vehicle-mountable computing platforms for autonomous control13–15 must be equipped with parallel and programmable coprocessors, such as embedded FPGAs, GPUs, neural processing unit (NPUs), to efficiently execute diverse and computationally intensive workloads. Among various types of Ising machines, some19–22,41–43 are based on highly parallelizable algorithms that are not limited to specialized hardware and do not require special equipment such as lasers or dilution refrigerators, and thus can potentially be implemented and accelerated with vehicle-mountable parallel coprocessors. These embeddable Ising machines may enable more rational judgments and functional information processing based on NP-hard combinatorial optimization for automated control. Some studies47–50 have investigated the applicability of quantum mechanics-based Ising machines (quantum annealers) to assignment problems in MOT under the assumption of one-to-one correspondence. Other studies have reported centralized (out-of-vehicle) systems using quantum annealers for traffic flow optimization51 or swarm robot control52. High-speed financial trading systems using Ising machines have also been demonstrated53–55. However, vehicle-mountable systems using Ising machines for autonomous control have not yet been studied or demonstrated.

To demonstrate the potential and feasibility of enhancing vehicle-mountable control systems using emerging Ising machines, we propose and implement a real-time MOT system featuring an enhanced temporal association mechanism, called the flexible assignment function, which is based on NP-hard combinatorial optimization and enabled by an embedded Ising machine.

With this flexible assignment mechanism, the proposed system enables robust object tracking through multiple long-term occlusion events. The assignment problem between detections and tracks is formulated as a QUBO problem, whose total cost function is a linear combination of an objective function that maximizes overall likelihood and a penalty function corresponding to the constraint for one-to-one correspondence. The system solves the QUBO problem twice per video frame while changing the weight coefficient for the penalty function (i.e., adjusting the strictness for one-to-one correspondence) and then detects the occlusion events and their locations as the difference between the two solutions (i.e., the two assignment tables), where an assignment table with many tracks being matched to one detection (a constraint-violation solution) may be selected if it is more plausible in terms of the total cost function upon the execution with the small weight coefficient. The QUBO formulation for the flexible assignment is related to but distinct from the time-series assignment QUBO recently proposed in ref. 49. The difference between them is discussed.

The system employs an embeddable Ising machine based on a quantum-inspired algorithm known as simulated bifurcation (SB)19–25, which enables solving the QUBO twice per video frame while maintaining real-time throughput. The SB algorithm was originally derived in 201919 by classicizing a quantum-mechanical Hamiltonian describing a quantum adiabatic optimization method56 and was further improved in 202121. It numerically simulates the time evolution of a classical nonlinear oscillator network exhibiting bifurcation phenomena, where the two branches of bifurcation in each oscillator correspond to the two states of an Ising spin. The operational mechanism of SB, which consistently finds better solutions with higher probability, is based on an adiabatic and ergodic search19. The MOT system is implemented using two vehicle-mountable, mid-range FPGAs: one is for object detection and the other for the assignment by the SB-based Ising machine. We demonstrate a real-time system-wide throughput exceeding 20 frames per second along with the Ising machine-enhanced MOT functionality. To evaluate the tracking capability through long-term and complex occlusion events, a systematic set of benchmark sequences has been prepared and is provided as the Supplementary Movies and Data.

Results
Figure 1 presents the block diagram of the proposed real-time, vehicle-mountable MOT system incorporating an SB-based embedded Ising machine. The system manages a set of tracks, which are objects currently being identified and followed, and updates them frame by frame using detections, which are the objects detected in the current video frame. The correspondence between tracks and detections is determined by solving a flexible assignment problem, which is central to this work and is formulated as a QUBO problem. The MOT system is implemented on a vehicle-mountable computing platform designed to meet constraints on size, power, and cost. It achieves real-time processing speeds and demonstrates improved tracking performance compared with a baseline system that solves a conventional linear assignment problem. The following subsections describe, in order, the QUBO formulation of the flexible assignment function, the enhanced MOT algorithm, the system architecture and implementation, the experimental demonstration, and a comparison with an alternative QUBO-based assignment method.

Flexible assignment
A similarity matrix S (also referred to as an association matrix in other literature) is defined as a measure of distance in real and/or feature spaces for all pairs (t, d) between tracks and detections, with each matrix element S being a real number. Based on the similarity matrix, an assignment function subject to the constraint for one-to-one correspondence determines an assignment table, where each element b is a binary decision variable indicating either match or unmatch. The proposed flexible assignment function (Fig. 2), which uses two binary assignment tables, introduces a third state, potentially-match, to represent a track being matched with an occluded object (i.e., a hidden detection). The state of potentially-match is utilized to more accurately estimate the dynamics of tracks, as discussed in the next subsection.

The flexible assignment function enables the detection of occlusion events and their locations. In this subsection, we explain the QUBO formulation of the flexible assignment problem step by step. For details on the QUBO/Ising formulation and their conversion, refer to the Methods section. We begin with the formulation of the linear assignment problem (i.e., a bipartite graph matching problem) under the assumption of one-to-one correspondence. Given Nt tracks and Nd detections in a video frame, we define NtNd binary variables, {bt,d}, each representing whether tth track is matched with dth detection:

bt,d = 1, (if tth track and dth detection are matched)
0, (if tth track and dth detection are unmatched): (1)

As a combinatorial optimization problem, we search for a bit configuration {bt,d} (i.e., an assignment table) that minimizes the objective function Hobject, which corresponds to maximizing the overall likelihood:

Hobject = - sum_{d=1}^{Nd} sum_{t=1}^{Nt} St,d bt,d (2)

subject to the linear equality constraints enforcing one-to-one correspondence:

sum_{t=1}^{Nt} bt,d = 1 for all d
sum_{d=1}^{Nd} bt,d = 1 for all t (3)

where we assume Nt = Nd.

The assignment problem described in Eqs. (2) and (3) is a constrained binary optimization problem. To convert it into an unconstrained form (QUBO), we apply the penalty method described in the Methods section. This approach minimizes the total cost function Hcost, which is a linear combination of the objective function Hobject and a penalty function Hpenalty corresponding to the constraints for one-to-one correspondence:

Hcost = Hobject + lambda Hpenalty (4)

Hpenalty = sum_{d=1}^{Nd} (sum_{t=1}^{Nt} bt,d - 1)^2 + sum_{t=1}^{Nt} (sum_{d=1}^{Nd} bt,d - 1)^2 (5)

Here, lambda is a weight coefficient for the penalty function. Hpenalty is minimized to zero when the constraints of Eq. (3) are satisfied (i.e., each column and row of the assignment table contains exactly one). However, Hcost may still decrease if the reduction in Hobject outweighs the increase in Hpenalty. To ensure equivalence between the constrained and unconstrained formulations, Hcost must always increase when a constraint violation occurs, compared with a constraint-satisfying and Hobject-minimum bit configuration (i.e., an exact solution). The bit configuration shown in Fig. 2 satisfies the constraints and can therefore be considered an example of an exact solution. When a bit flips from zero to one in such a configuration, the resulting change in Hcost is given by (-St,d + 2lambda). This expression defines the critical condition for determining an appropriate value of lambda. Since the similarity score St,d is a relative measure, it can generally be normalized to the range [0, 1]. In this work, we use the Intersection over Union (IOU), denoted as IOU(t,d)4, as the similarity measure St,d (also see the next subsection), which by definition lies within [0, 1]. Therefore, the lower bound for the weight coefficient is lambda > 1/2.

In the flexible assignment function, we first extend the penalty function to accommodate one-to-zero and zero-to-one correspondences when Nt != Nd. In the case where Nt > Nd, tracks without corresponding detections (i.e., one-to-zero correspondence) are permitted. This situation typically arises when there are candidate tracks to be deleted, such as objects exiting the frame (i.e., frame-out objects). Conversely, when Nt < Nd, detections without corresponding tracks (i.e., zero-to-one correspondence) are allowed. This typically occurs when there are candidates for additional tracks (e.g., frame-in objects). However, any double coincidences (i.e., many-to-one correspondence) are treated as constraint violations.

The extended cost and penalty functions are defined as follows.

H = Hobject + lambda(Hpenalty1 + Hpenalty2) (6)

Hpenalty1 = sum_{d=1}^{Nd} (sum_{t=1}^{Nt} bt,d - 1)^2 (if Nt >= Nd)
sum_{t=1}^{Nt} sum_{d!=d0} bt,d bt,d0 (if Nt < Nd) (7)

Hpenalty2 = sum_{t=1}^{Nt} (sum_{d=1}^{Nd} bt,d - 1)^2 (if Nt <= Nd)
sum_{d=1}^{Nd} sum_{t!=t0} bt,d bt,d0 (if Nt > Nd) (8)

where sum_{t!=t0} (or d!=d0) represents the summation over NtC2 (or NdC2) pairwise combinations of bits in a column (or a row). Here, the constraints corresponding to the penalty function (Hpenalty1 + Hpenalty2) are explicitly expressed by

sum_{t=1}^{Nt} bt,d = 1 for all d (if Nt >= Nd)
sum_{t=1}^{Nt} bt,d = {0, 1} for all d (if Nt < Nd) (9)

sum_{d=1}^{Nd} bt,d = 1 for all t (if Nt <= Nd)
sum_{d=1}^{Nd} bt,d = {0, 1} for all t (if Nt > Nd) (10)

The penalty function, Hpenalty1 + Hpenalty2, is minimized to zero when the constraints of Eqs. (9) and (10) are satisfied. Suppose the case of Nt > Nd. If sum_{d=1}^{Nd} bt,d = 1 for the tth row in Eq. (10), then sum_{d!=d0} bt,d bt,d0 for the same row in Eq. (8) is 0 (no penalty) since a non-zero bit is paired with a zero bit in any pairwise combinations. In contrast, if sum_{d=1}^{Nd} bt,d = 2 (a constraint violation), then sum_{d!=d0} bt,d bt,d0 in Eq. (8) is 1 (a penalty for many-to-one correspondence) since both bits of one of pairwise combinations are 1. In contrast, if sum_{d=1}^{Nd} bt,d = 2 (a constraint violation), then sum_{d!=d0} bt,d bt,d0 in Eq. (8) is 1 (a penalty for many-to-one correspondence) since both bits of one of pairwise combinations are 1.

Let us consider the critical condition for determining lambda in Eq. (6). The bit configuration illustrated in Fig. 2h can be regarded as an example of an exact solution when Nt > Nd. Compared to this exact solution, if a bit flips from zero to one, as shown in the example in Fig. 2g, it induces a change of (-St,d + lambda) in Hcost. This represents the minimum possible change in Hcost, and it must be greater than zero. Hence, for the flexible assignment function, the bound condition for the weight coefficient is lambda > 1, assuming Smax = 1. If lambda is set to less than 1, a bit configuration that includes double coincidences (i.e., many-to-one correspondences) may be selected, since the bit flip corresponding to such a double coincidence could result in a decrease in Hcost.

To implement the flexible assignment function illustrated in Fig. 2a, we solve the QUBO problem twice per video frame (i.e., per similarity matrix) using two different weight coefficients: a large lambda (lambdalarge) and a small lambda (lambdasmall). These are processed using an Ising machine (a heuristic solver), resulting in two assignment tables. The table generated with lambdalarge is more likely to satisfy the constraints in Eqs. (9) and (10), whereas the one generated with lambdasmall is more likely to tolerate many-to-one assignments (i.e., constraint violations). We then arbitrage between the two potentially different tables to produce a final assignment result. First, we determine the state of each track as either match or unmatch based on the table for lambdalarge. Then, for the unmatched tracks, those that have corresponding detections in the table for lambdasmall are labeled as potentially-match. In this work, we use 1.0 and 0.1 for lambdalarge and lambdasmall, respectively. The value of lambdalarge is chosen based on the bound condition. Although lambdalarge > Smax (= 1) is required, we set lambdalarge = 1.0 because the occurrence of St,d = 1 is rare. This choice is validated through an ablation study for lambdalarge (see the Methods section). The value of lambdasmall is similarly determined through an ablation study for lambdasmall (also detailed in the Methods section).

Figure 2c, f illustrate two successive frames: one showing a scene with one-to-one correspondence between tracks and detections (Nt = Nd), and the other depicting an occlusion event (Nt > Nd) where the object tracked as track ID=5 is occluded by (i.e., not detected due to) the object tracked as track ID=2. For the scene in Fig. 2c, the panels in Fig. 2d, e show the resultant assignment tables for lambdasmall and lambdalarge, respectively, in the format of arranging a bit configuration {bt,d} in an Nt x Nd matrix. Fig. 2g, h present the corresponding tables for the scene in Fig. 2f.

For the scene in Fig. 2c (without occlusion), the assignment tables in Fig. 2d, e for lambdasmall and lambdalarge are identical and satisfy the constraint for one-to-one correspondence (i.e., all the sums of bt,d in rows/columns are 1). In contrast, for the scene in Fig. 2f (with occlusion), the assignment table in Fig. 2h for lambdalarge satisfies the constraint. Based on this table, track ID=5 is initially determined to be unmatch by the arbiter (depicted in Fig. 2a). At this stage, two possibilities exist for the object tracked by track ID=5: either it has exited the scene (frame-out or left behind buildings) or it is temporarily occluded (crossing). The assignment table in Fig. 2g for lambdasmall, however, violates the constraint, as both tracks ID=5 and ID=2 are matched with detection ID=4. Such a solution may be selected by the Ising machine when the reduction in the objective function, due to more matched tracks, outweighs the increase in the penalty function caused by constraint violations. Based on the table in Fig. 2g, the arbiter finally determines the state of track ID=5 to be potentially-match. Thus, the system detects both the occurrence and the location of an occlusion event.

Figure 2b illustrates the solution space of the decision variables (i.e., all the possible bit configurations) including both constraint-satisfying and constraint-violating solutions. Assuming Nt = Nd (= No) for simplicity, the size of the solution space is 2^(No^2), while the number of solutions that satisfy the one-to-one correspondence constraint is No!, which constitutes only a small fraction of the total space. The bit configurations shown in Fig. 2d, e, h belong to the constraint-satisfying subset, whereas the configuration in Fig. 2g belongs to the constraint-violating subset. The linear assignment algorithm (e.g., the Hungarian algorithm) searches exclusively within the subset of constraint-satisfying solutions. In contrast, the Ising machine explores the entire solution space, including configurations that violate constraints. This broader search capability is essential for identifying plausible many-to-one assignments. It is crucial to evaluate the degree of constraint violation using the penalty functions (the quadratic functions), represented by the quadratic terms in Eqs. (7) and (8), rather than binary judgments imposed by the equality constraints (Eqs. (9) and (10)).

Multiple object tracking
Many modern MOT systems4–10 rely on the Kalman filter framework57 to estimate the motion dynamics of tracks. In this framework, the states of tracks are first predicted based on a motion model at each frame and then corrected (updated) using the information from matched detections (i.e., measurement results) at that frame (see also Fig. 1 and Fig. 2a). This approach enables more accurate motion estimation by incorporating a series of measurements observed over time rather than relying on a single measurement, which may be affected by statistical noise or other inaccuracies.

Figures 1 and 2a show the block diagram of the MOT system, which consists of a camera, a detector, a predictor, a corrector, an associator, and an assignor. The pseudocode in Algorithm 1 (see the Methods section) outlines the information processing procedure within the system. In this work, we adopt the Simple Online and Real-time Tracking (SORT) system4 as a baseline because it is a representative example of modern MOT systems and is simple to implement on vehicle-mountable computing resources. We then modify it in two key aspects: First, we replace the assignment function in the assignor. Instead of using the linear assignment method that assumes one-to-one correspondence (i.e., the Hungarian algorithm), we introduce the flexible assignment method using an Ising machine and an arbiter (see Fig. 2a). Second, we modify the procedure in the corrector to incorporate the newly introduced potentially-match state, which enables robust tracking through occlusion events.

The status data of ith track include a vector Ti:

Ti = {ri, r_i, agei} (11)

where ri represents the location and size of the bounding box for the track, r_i is the time derivative of ri, and agei is the age of the track, defined as the number of frames (i.e., the time) elapsed since it last acquired a matched detection. At the beginning of the procedure for each frame (see Algorithm 1), the statuses of tracks are predicted by approximating the inter-frame displacements of objects, and the age of all tracks are incremented by one. At the end of the procedure, tracks whose age exceeds a predetermined lifetime, max_age, are deleted. The processing for each track in the corrector depends on the assignment result (match, potentially_match, or unmatch) determined by the assignor. The assignor determines the assignment result based on the predicted tracks and the detections obtained at the current frame. A matched track is updated using the corresponding detection (based on the Kalman filter theory), and its age is reset to zero. A potentially-matched track is not updated (its predicted status is retained for the next frame), but its age is decreased by a predetermined constant, anti_aging. An unmatched track is left unchanged and is deleted if its age exceeds max_age. Additionally, a new track is appended for each unmatched detection (if exists). In this work, both max_age and anti_aging are set to 5.

Once a track is determined to be in the potentially_match state, it is not deleted for at least subsequent anti_aging frames. Furthermore, if it is again classified as potentially_match again during the period, its age is further decreased by anti_aging (allowing the age to become negative). This mechanism enables the track to re-establish correspondence with a detection after an occlusion event, which typically lasts for several frames. These special treatments are applied only to specific tracks, namely, those in the potentially-match state. In contrast, tracking through occlusion could also be achieved by simply increasing max_age without introducing the potentially_match state. However, this approach has significant drawbacks. In such cases, tracks corresponding to objects that should be deleted (e.g., frame-out objects or those left behind buildings) may persist for a long period. This leads to unnecessary computational overhead in managing an inflated number of tracks within the Kalman filter framework and increases the risk of erroneous match between those unfavorable tracks and detections. For a quantitative verification of this discussion, see the ablation study for max_age in the Supplementary Information 1.

System architecture
To demonstrate the feasibility of the proposed MOT system under constraints of size, power, and cost for vehicle-mountable computing platforms, we prototyped the system on two vehicle-mountable computing boards. Each board is equipped with an embedded FPGA (serving as a parallel and programmable coprocessor) and a general-purpose microprocessor unit (MPU). Figure 3 shows the hardware configuration of the system and indicates where the modules depicted in Figs. 1 and 2a are implemented. Among these modules, the computationally intensive components are the Ising machine and detector, which are hardwired (i.e., implemented as custom circuits) on the two FPGAs to ensure efficient processing in terms of speed and power. The remaining modules are implemented as software objects and executed on the MPUs.

The Ising machine used in this work supports a 512-spin configuration with all-to-all spin-spin connectivity, allowing real numbers (32-bit precision) to be set in any coupling coefficients (Ji,j). The QUBO problem (with decision variables, bi in {0,1}) defined by Eqs. (1) and (6) is represented as an Ising problem (with decision variables, si in {-1,1}) and then solved using the Ising machine. See the Methods section for details on the one-to-one correspondence between QUBO and Ising formulations. The current implementation supports up to 22 tracks (Nt=22, Nd=22, NtNd=484).

SB is a highly parallelizable metaheuristic algorithm for solving discrete optimization problems. For N-spin Ising problems with full connectivity, the maximum numbers of parallelizable operations in SB and simulated annealing (SA, a conventional metaheuristic)58,59 are, respectively, N^2 and N20,22. Custom-circuit implementations of SB20,22,24 on modern FPGAs with island-style architectures60 have demonstrated a degree of computational parallelism exceeding the problem size N. Among the various SB variants, we adopt the ballistic SB algorithm21 in this work, as it is well-suited for single-shot processing necessary for high-speed real-time systems53–55. See the Methods section for further details on SB.

Using a scalable design of the accelerator for ballistic SB, written in a high-level synthesis language and based on circuit architectures similar to those in Refs. 20,61, we built the embedded 512-spin SB-based Ising machine shown in Fig. 3c. It features 2048 parallel processing elements (PEs). These 2048 PEs compute 2048 pair interactions simultaneously in a single clock cycle. These interactions are part of total 512x512 interactions to be calculated per SB timestep (corresponding to the term of sum_{j} Ji,j xj in Eq. (20)).

The degree of computational parallelism was chosen based on estimated cost constraints for commercial vehicle applications: the number of logic elements was kept below 250K, and the number of 32-bit digital signal processor (DSP) units below 400. When Nstep, an operational parameter for SB, is 400 (the case for this work), the time required to obtain a solution with the SB-based Ising machine is 284 μs. Since the QUBO problem is solved twice per frame (see Flexible assignment subsection), the total computation time per frame is 568 μs. The operating power of the Ising machine during real-time operation of the MOT system was measured to be 3.4 W. See the Methods section for further details on the implementation of the MOT system.

Demonstration
The proposed MOT system, equipped with an embedded Ising machine and implemented on vehicle-mountable computing boards, demonstrates real-time processing speed and enhanced tracking capability through NP-hard (quadratic) combinatorial optimization, in contrast to the baseline system that relies on linear combinatorial optimization (i.e., the original SORT4). In addition to standard benchmark sequences from the MOT challenges62,63, we have prepared a systematic set of benchmark sequences designed to evaluate tracking performance under long-term and complex occlusion events. These sequences were generated using CARLA simulator (MIT License) and assets (CC-BY License)64, which are provided as the Supplementary Data 1.

We first demonstrate that the proposed system achieves real-time processing speed, with the calculation time of the embedded Ising machine being minor compared to the overall processing time. Figure 4a shows a histogram of the computation times for the modules in the MOT system when processing a benchmark video sequence (600 frames), titled “MOT17-02-FRCNN”62,63. Figure 4b illustrates the tracks as colored bounding boxes in a scene extracted from the same sequence. The inset in Fig. 4a presents a timing chart of the MOT system. In this chart, the operation of the detector (Tdetector) is overlapped with that of the other tracking modules (Ttracking) including the assignor (Tassignor). The processing time per frame is determined by max{Tdetector, Ttracking}, which is 44.2 ms on average, corresponding to a processing speed of 23 frames per second. The computation time of the embedded Ising machine (TIsing_machine) is deterministic (568 μs) and minor compared to (or not affecting) the overall processing time. Note that Tassignor is measured by the MPU on the left board in Fig. 3, and includes the time for inter-board communication as well as the computation times for the preprocessor and the arbiter. See the Methods section for further details on system-wide throughput. The detector receives video frames either from onboard memory or directly from the camera. See the Supplementary Movie 1 for a demonstration of real-time operation using live camera input. The authors affirm that all participants appearing in the Supplementary Movie 1 have provided informed consent for the publication of the images.

The proposed MOT system enables robust tracking through long-term and complex occlusion events, a capability that cannot be achieved with the baseline system. To demonstrate this, we designed a custom video sequence (the Supplementary Movie 2) featuring simultaneous occurrences of three-object and two-object crossings. Here, the three-object crossing is involved in the long-term two-object occlusion event, where two objects move in the same direction, and one overtakes the other at a small relative speed. The proposed system correctly tracks the five objects through those occlusion events, while the baseline fails to track. A comparative video showing the tracking results of both systems is provided in the Supplementary Movie 2. Figure 4c–e show three scenes before, during, and after the complex occlusion event. During the simultaneous occurrences of three-object and two-object crossings, the states of tracks ID=0, 1, 4 are potentially-match as shown in Fig. 4g.

The proposed system is capable of tracking through even more complex occlusion events. The Supplementary Movie 3 provides an example with simultaneous occurrences in four locations of five-object crossings.

To quantitatively evaluate the enhancement in tracking performance of the proposed MOT system, we compare it to the baseline using a common parameter of max_age (= 5), across a series of benchmark sequences that are assessed using an MOT evaluation metric called HOTA (higher order tracking accuracy)65,66. HOTA is a unified and balanced metric comprising three sub-metrics: AssA (association accuracy), DetA (detection accuracy), and LocA (localization accuracy). While many recent MOT papers use the MOT challenges datasets4–7,9–11,49, these benchmark sequences are not specifically designed to evaluate various occlusion scenarios, and the occurrence of occlusion events is moderate. Therefore, we designed nine benchmark sequences that include complex crossing events (provided with ground truth data in the Supplementary Data 1) to evaluate tracking performance under long-term and complex occlusion conditions, where the frequency of occlusion events is systematically varied. See the Methods section for details on the benchmark sequences.

Table 1 summarizes the measured results for seven benchmark sequences, MOT17-{02, 04, 05, 09, 10, 11, 13}-FRCNN, from the MOT challenges, and the nine benchmark sequences for crossing, Cross-{DLVL, DLVM, DLVH, DMVL, DMVM, DMVH, DHVL, DHVM, DHVH}, where the numbers (percentages) in parentheses indicate the relative improvements over the baseline. Movies comparing the tracking and assignment results between the proposed MOT system and the baseline, when processing the Cross benchmark sequences, are provided as Supplementary Movies 4 to 12. The overall HOTA score is improved for the proposed MOT system, primarily due to enhancements in AssA achieved through the flexible assignment function, rather than improvements in detection and localization accuracies (DetA and LocA). The improvements in HOTA for the proposed MOT system tend to be more significant for benchmark sequences that include a higher number of occlusion events. See the ablation studies in the Methods section and Supplementary Information 1 for further details on comparison between the proposed system and baseline.

QUBO-based assignment
The flexible assignment function proposed in this work is realized by solving the QUBO problem defined with Eqs. (6) to (8). While many MOT systems4–11,50 rely on linear assignment, Zaech et al. recently introduced a QUBO-based (quadratic) assignment for MOT49, which is theoretically grounded in Refs. 47,48. Below, we describe how Zaech’s QUBO differs from ours.

Unlike our QUBO, which is intended for online (or real-time) processing, Zaech’s QUBO is designed for offline (or batch) processing. Specifically, our MOT system updates tracks frame by frame, while Zaech’s MOT system determines plausible assignments between tracks and detections over a time span (for a batch of frames). Hereafter, our QUBO and Zaech’s QUBO are referred to as the flexible assignment QUBO and the time-series assignment QUBO, respectively.

For the time-series assignment QUBO, we prepare F assignment tables, each consisting of NtNd binary variables, where F is the batch size (i.e., the number of frames in a batch), and Nt and Nd are the numbers of tracks and detections, respectively. Thus, the QUBO involves a total of F NtNd binary variables. Similarity is defined for pairs of detections across different frames, and the objective function includes a quadratic term of the form St,di,dj bt,di bt,dj, where i and j are frame indices, S represents the similarity between detections di and dj, and bt,di is a binary variable defined as:

bt,di = 1, (if tth track and dth detection at ith frame are matched)
0, (if tth track and dth detection at ith frame are unmatched): (12)

The quadratic term represents a gain when dth detection at ith frame and dth detection at jth frame are assigned to the same tth track (i.e., when bt,di bt,dj = 1). By solving the QUBO under the constraint of one-to-one correspondence within each frame, we can simultaneously determine the F assignment tables, thereby constructing the trajectories over time.

In the flexible assignment QUBO, the quadratic terms for pairwise combinations of binary variables (Eqs. (7) and (8)) are evaluated to quantify the degree of constraint violations related to one-to-one correspondence, thereby enabling plausible many-to-one assignments. To further enhance tracking accuracy, it may be possible to incorporate the quadratic term from the time-series assignment QUBO into the objective function of the flexible assignment QUBO (Eq. (2)). This integration is left for future work.

Methods
QUBO and Ising problems
The QUBO and Ising problems are mathematically equivalent and classified as nondeterministic polynomial-time hard (NP-hard) problems45,46. Many NP-hard and NP-complete problems, including all of Karp’s 21 NP-complete problems, can be formulated as QUBO or Ising problems46. The N-variable QUBO problem is to find a bit configuration (from among 2^N candidates) that minimizes the cost function:

HQUBO(b) = b^T Q b = sum_{i,j} Qij bi bj (13)

where bi (in {0, 1}) denotes ith bit, b = (b1, ..., bN) is the vector representation of a bit configuration, Qij (= Qji) is a quadratic coefficient for ith and jth bits, Q is the matrix representation of {Qij}. Since bi^2 = bi, the diagonal elements Qii represent linear coefficients for bi. The N-variable Ising problem is to find a spin configuration that minimizes the Ising energy45:

HIsing(s) = - 1/2 sum_{i=1}^{N} sum_{j=1}^{N} Ji,j si sj + sum_{i=1}^{N} hi si = - 1/2 s^T J s + h^T s (14)

where si (in {-1, 1}) is ith Ising spin, s = (s1, ..., sN) is the vector representation of a spin configuration, Ji,j (= Jji) is a coupling coefficient between ith and jth spins (Ji,i = 0), J is the matrix representation of {Ji,j}, and hi is a bias (or linear) coefficient for ith spin, and h is the vector representation of {hi}.

The QUBO problem in the form of Eq. (13) can be written as an equivalent Ising problem in the form of Eq. (14) using the following relations:

si = 2bi - 1 (15)

Ji,j = 2 Qij (if i != j)
0 (if i = j) (16)

hi = sum_{j} Qij (17)

Penalty method
To solve constrained binary minimization problems using (unconstrained) QUBO (or Ising) solvers, a penalty method46,47,49 is often employed. This method reformulates a constrained problem as an unconstrained one, with the solution ideally converging to that of the original constrained problem. The unconstrained problem is created by adding a term called a penalty function to the original objective function.

A quadratic bit minimization problem with linear equality constraints (Gb=d) is generally expressed as:

argmin_{b} b^T Q b s.t. Gb=d (18)

where G and d are the coefficient matrix and constraint vector, respectively. This constrained problem is transformed into the following unconstrained problem:

argmin_{b} b^T Q b + lambda ||Gb - d||^2 (19)

where the second term, ||Gb - d||^2, serves as the penalty function. Here, lambda is a weight coefficient (a positive constant) for the penalty function, and the Frobenius norm is defined as ||A||F = sqrt(sum_{i,j} Aij^2) (i.e., ||A||^2F = sum_{i,j} Aij^2). The penalty function is minimized to zero if the equality constraints are satisfied (i.e., if Gb=d); otherwise, it increases quadratically depending on the degree of constraint violation. The expression of Eq. (19) is quadratic and unconstrained (it can be reformulated in the form of Eq. (13)), and thus can be solved using QUBO(/Ising) solvers.

The penalty weight lambda is an important hyper-parameter that must be appropriately determined depending on the problem. To ensure the constraints are satisfied, lambda needs to be large. However, an excessively large lambda creates a steep energy landscape (HQUBO/Ising), which potentially destabilizes the dynamics within Ising solvers. Therefore, in practice, lambda should be as small as possible while still ensuring equivalence between Eqs. (18) and (19).

Simulated bifurcation
SB19,21 is a quantum-inspired19,68, highly parallelizable20,22,24, metaheuristic algorithm for computationally hard combinatorial (or discrete) optimization. SB-based Ising machines belong to a class of oscillator-based Ising machines27–29,34–37,43. The SB algorithm finds optimal (exact) or near-optimal solutions to the Ising problem by simulating the time-evolution of coupled nonlinear oscillators according to Hamilton’s equations of motion (without energy-dissipative or noise-based mechanisms). Several variants of SB exist, including adiabatic SB, ballistic SB, and discrete SB, which differ in nonlinearity69 and discreteness21.

In the SB algorithm, the ith nonlinear oscillator corresponds to the ith Ising spin and its state is described by its position and momentum (xi, yi). The update procedure for xi and yi in the ballistic SB, used in this work, is as follows21:

yik+1 = yik + (a0 - atk) xik - eta hi + c0 sum_{j} Ji,j xj^k Delta t (20)

xik+1 = xik + a yik+1 Delta t (21)

(xi, yi) = (sgn(xi), 0) (if |xi| > 1)
(xi, yi) (if |xi| <= 1) (22)

where a0, c0 and eta are positive constants, atk is a control parameter increasing from zero to a0, and sgn(x) (equal to +/- 1) is the sign function. Eq. (22) represents a nonlinear transfer function69, which physically corresponds to a perfectly inelastic wall at x = +/- 1. The time increment is denoted as Delta t, such that tk+1 = tk + Delta t. After iterating this update procedure for a predetermined number of timesteps (Nstep), the final ith position xi is binarized to yield the ith spin (+/- 1) by taking the sign of xi. In this work, a0 = 1, c0 = eta = 0.8, Delta t = 0.3, and Nstep = 400.

The ballistic SB algorithm has been demonstrated to produce higher-quality solutions more efficiently than the simulated annealing (SA) algorithm, for both academic benchmark problems21 and practical applications25,53–55,61.

MOT algorithm
Algorithm 1 outlines the processing step in the MOT system with flexible assignment, comprising the following components: camera, detector, predictor, corrector, associator, and assignor, as illustrated in the block diagrams in Figs. 1 and 2a.

The detector detects objects of the “car” or “person” classes in each frame using a real-time object detection algorithm, YOLO70,71 and outputs the corresponding detections (including the bounding boxes for detected objects). Similar to SORT4, the predictor is based on a linear constant velocity model, and the associator uses the Intersection over Union, IOU(t, d), of the bounding boxes for tth track and dth detection as the similarity measure St,d. The assignor and corrector are described in detail in the Main text.

Implementation
To implement the proposed MOT system, we used two vehicle-mountable SoC (System-on-a-Chip)-FPGA boards, each equipped with a monolithic MPU-FPGA chip.

The first board is the Intel Arria 10 SX SoC Development Kit (DK-SOC-10AS066S-D), featuring a 10AS066N3F40E2SG1 monolithic chip that integrates a dual-core ARM Cortex-A9 MPCore processor and an embedded FPGA. The FPGA (660K logic elements, 4-input LUT equivalent) has 251,680 adaptive logic modules (ALMs) including 251,680 adaptive lookup tables (ALUTs, 6-input LUT equivalent) and 1,006,720 flip-flop registers, 213 120 Kbit-size RAM blocks (BRAMs)72, and 337 418-bit x 19-bit multipliers (DSPs).

The SB-based Ising machine was implemented with this FPGA using a high-level synthesis (HLS) language (Intel FPGA SDK for OpenCL, ver. 18.1). Table 2 summarizes the architecture and the implementation results. The system clock frequency (Fsys) after synthesis, placement, and routing is 254 MHz. The operating power of the Ising machine is 3.4 W, as measured by Power Monitor tool that uses the onboard MAX V CPLD to monitor current on the FPGA power rails. Software components executed on the MPU were written in C/C++ programming language and ran on the Angstrom Linux OS (v2014.12).

The second board is the AMD Zynq UltraScale+ MPSoC (ZCU104), which features a XCZU7EV-2FFVC1156 monolithic chip including a quad-core ARM Cortex-A53 MPCore processor and an embedded FPGA. The FPGA (504K logic cells, 4-input LUT equivalent) has 28,800 configurable logic blocks (CLBs) including 230,400 adaptive lookup tables (ALUTs, 6-input LUT equivalent) and 460,800 flip-flop registers, 312 36 Kbit-size RAM blocks (BRAMs)/96 288 Kbit-size RAM blocks (UltraRAMs), and 1728 27-bit x 18-bit multipliers (DSPs).

The detector was implemented on this FPGA as a custom circuit for YOLOv271, yolov2_voc_pruned_0_77, provided by Advanced Micro Devices, Inc. Software components for the MPU were written in Python (version 3.8) and executed on Ubuntu Linux OS (20.04.5/6 LTS).

Both boards are equipped with Ethernet Media Access Controllers (EMACs) and are interconnected via a 1 Gbps Ethernet cable using the UDP protocol.

Benchmark sequences for crossing
Table 3 lists nine benchmark sequences designed to evaluate crossing scenarios. Each sequence is characterized by the number of frames (#frames), the maximum and average number of objects (#objects), the average object velocity (in meters per second, assuming 20 FPS), and the average occlusion ratio. In the ground truth data, objects are labeled as either visible or invisible (occluded). The occlusion ratio for a given frame is defined as (number of objects - number of visible objects)/(number of objects).

Ablation study for lambdalarge and lambdasmall
The two weight coefficients, lambdalarge and lambdasmall, are important hyper-parameters in the proposed MOT system. Figure 5a, b show the variation in the overall HOTA score (MOT17+Cross) when (a) both lambdalarge and lambdasmall are varied together and when (b) lambdasmall is varied while keeping lambdalarge = 1.0, respectively.

The value of lambdalarge was set to 1.0 based on the bound condition described in the Main text. When increasing lambdalarge (= lambdasmall) toward 1.0, the overall HOTA score improves and then saturates near lambdalarge = 1.0, the bound condition. The proposed MOT system with lambdalarge = lambdasmall = 1.0 is substantially the same as the baseline (the original SORT4). In fact, the overall HOTA scores for the proposed MOT system with lambdalarge = lambdasmall = 1.0 and the baseline are 50.24 and 50.21, respectively (both are almost identical).

As shown in Fig. 5b, when decreasing lambdasmall from 1.0 to 0.0 while keeping lambdalarge at 1.0, the overall HOTA score increases (indicating enhancement compared to the baseline) and peaks at lambdasmall = 0.1. Therefore, lambdasmall was determined to 0.1 in this work. Note that if lambdasmall is too small (< 0.1), unnecessary tracks in potentially-match states may be generated, leading to erroneous associations.

System-wide throughput
Table 4 lists measured system-wide throughput when processing the benchmark sequences (Cross+MOT17) with the prototyped vehicle-mountable system shown in Fig. 3. The baseline does not use the right board and runs the linear assignment function (i.e., the Hungarian algorithm) with the MPU on the left board. As stated in the Main text and shown in the inset of Fig. 4a, the processing time of the embedded Ising machine (TIsing_machine) is 568 μs and is not a limiting factor in determining the overall processing time. The overall processing time is mainly determined by the time to manage tracks in the Kalman filter framework, and thus the processing time increases with the number of tracks required (depending on the benchmark sequences). The additional time components observed for the proposed system are due to the inter-board communication time and the computation times of the preprocessor and the arbiter (with the MPU on the right board), which also depend on the number of tracks. To minimize these additional times, all system components should be integrated into a single SoC board. Processing time for MOT17-{04, 10, 11, 13}-FRCNN is unavailable for the proposed system because the required number of tracks exceeds the Ising machine’s size limitation (see System architecture subsection). The accuracy data in Table 1 for these sequences were obtained with a similar but larger (2048-spin configuration) Ising machine61.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The authors modeled the multi-object-tracking (MOT)'s assignment problem as a Quadratic Unconstrained Binary\nOptimization (QUBO). The MOT task is addressed through a two-stage minimization process of the proposed QUBO, where\nthe second stage focuses on refining the initial penalty factors. The paper presents an innovative use of Ising machine on a\nmountable computing platform. While the implementation and experimental results are interesting, major concerns should be\naddressed.\n1. What is the novelty of the proposed Ising/QUBO compared to\nZaech, J. N., Liniger, A., Danelljan, M., Dai, D., & Van Gool, L. (2022). Adiabatic quantum computing for multi-object tracking.\nIn Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8811-8822).\nThey also proposed QUBO to tackle MOT, and since a QUBO can be turned into an Ising model, their QUBO is solvable by\nyour Ising Machine. Therefore, the manuscript lacks discussion/comparison on the above QUBO model. Your paper would\nbe much more interesting if the two models could be compared in terms of qubit/spin complexity (the number of variables in\nthe models given Nt and Nd), running time, MOT metrics, and so on. A section is needed for the above paper to be fully\ndiscussed. You also need to show the gap that why your model is needed at all and why not using their model on your Ising\nmachine.\n2. The following paper is closely related to the paper. It should be added in the literature part for interested readers.\nBirdal, T., Golyanik, V., Theobalt, C., & Guibas, L. J. (2021). Quantum permutation synchronization. In Proceedings of the\nIEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13122-13133).\n3. The penalty factor “c” in Eq.2: The authors should provide a bound for this parameter. How to select a value for “c” to make\nsure the constraints defined in Hpenalty1 and Hpenalty2 are satisfied? This should be justified mathematically. The state\n“larger c as 1.0 and smaller c as 0.1” is not sufficient and acceptable.\n4. “(Hpenalty1 + Hpenalty2)=0 if there are no violations”. This should be proved mathematically for “if and only if.”\n5. To make the QUBO explanation clearer, it is better to add a simple example for a few numbers of trackers and detections,\nand exemplify how the second part of the Eq.2 ensures the one-to-one correspondence (maybe when Nt>Nd or Nt<Nd).\n6. “The table for the larger c more likely satisfies the constraint for one-to-one correspondence, while one for the smaller c\nmore likely tolerates many-to-one assignments (constraint violations):” This state should be proved mathematically or\njustified based on an ablation study.\n7. Overall, the QUBO needs mathematical proof to show the corresponding minimum-cost state provides the optimal solution\nto the assignment problem. This has to be in general, not by examples.\n8. The two-stage minimization must be justified. Based on what metric means, the authors guarantee that by having a two-\nstage minimization, the optimal solution to the initial assignment problem can be obtained/estimated by the Ising solver.\n9. Table 2 needs running time as well\n10. What are the limitations of the proposed MOT system, and what can be done in future research to improve it?\n11. Overall, the paper is not easy to follow (which becomes even worse for unfamiliar readers to Ising/QUBO OR\nMOT). The QUBO formulation should be in the Methods Section. You have a mix of everything in each section\n12. The discussion/conclusion section must be extended.", "reviewer_2": "The paper presents a novel flexible assignment function for existing multi-object tracking methods based on Ising machines.\nUnlike existing methods it is not limited to linear one-to-one assignments which can be used to detect and handle occlusions\nto prevent id-switches.\nWhile the presented experimental evaluation and the supplementary material shows promising results, there are several\nmajor concerns regarding the experimental study and the manuscript itself that need to be addressed before the paper is\nready for publication.\nGeneral comments:\n- I suggest to use the term „tracks“ instead of „trackers“ to avoid confusion with tracking algorithms.\n- Sec. 2 describes the proposed method, so I suggest to use another name than „Results“\nStrengths:\n- The supplementary videos are helpful in understanding the benefits of the proposed approach.\n- Table 2 convincingly emphasises that the flexible assignment marginally outperforms the Hungarian matching on 7 hand-\npicked movies\nWeaknesses:\n- In general, the paper seems to be hard to understand for readers unfamiliar with Ising machines and Muti-object tracking\nand is thus not inappropriate to target a broader audience.\n- There are many spelling and grammar errors such as missing plurals or articles as well as syntax errors. Many paragraphs\ndo not connect well to the respective prior paragraph making the overall story of the sections hard to follow. For instance, the\nintroduction combines paragraphs on multi-object tracking, Ising machines, embedded systems without transitions that help\nto guide the reader. Thus I believe that the manuscript needs major revision and is not ready for publication yet.\n- The evaluation is very limited.\n- c is set to 0.1 and 1.0 . A transparent discussion to show how sensitive the approach is to this hyperparameter would be\nhelpful.\n- The authors argue that their method based on „potential matches“ is better than extending the max-age of existing\nMOT methods. It would be good to also show this qualitatively.\n- An ablation for the parameters of the Ising machine would be helpful in understanding the proposed methods’\nlimitations", "reviewer_3": "This manuscript aims to present an in-vehicle multiple-object tracking system with an embeddable\nHowever, there are many issues with this manuscript.\n1) The contributions are unclear.\n2) The research issues and the response methods are unclear. The proposed method is weak.\n3) The experiments that were conducted did not show remarkable results. Current results are weak and can not support the\nproposed system and the claimed goal.\n4) The manuscript claims an in-vehicle multiple-object tracking system, however, the proposed system has not been\ndemonstrated on well-known driving datasets such as nuScenes. Furthermore, the proposed system claims a real-time in-\nvehicle MOT system; however, the system has not been demonstrated in a vehicle system and driving scenarios. Instead,\nan embedded Ising machine was used to test the system, which can not demonstrate the contributions of the proposed in-\nvehicle system.\n5) The structure of this manuscript is confused. The research goal, proposed solution, experiments, and results must be\ndesigned consistently.\n6) The writing in this manuscript needs to be significantly improved."}, "author_response": "Response to Reviewers:\nWe appreciate the time and effort that the Reviewers dedicated to reviewing the manuscript. We received many\nfruitful suggestions and comments from our three reviewers, based on their deep understanding, which led us to add\nsubstantial discussion and additional work. This has significantly improved the manuscript, making it much stronger,\nmore readable, and accessible to the broad readership of Nature Communications.\nSummary of Main Revisions:\n1) The Introduction and Discussion (Conclusion) sections have been revised to better describe the problem, objective,\nproposed method, and experiment.\n2) The proposed QUBO for flexible assignment, the core of this work, has been better explained. Firstly, to provide\nbackground information to readers, the \"QUBO and Ising Problems\" and \"Penalty Method\" subsections have been,\nrespectively, revised and added to the Methods section. Secondly, the \"Flexible Assignment\" subsection has been\nrevised to mathematically better explain the QUBO formulation of the flexible assignment function. Thirdly, the\n\"QUBO-based Assignment\" subsection has been added to compare the proposed QUBO with another recently\nproposed QUBO for time-series assignment.\n3) The Demonstration section has been substantially strengthened. Firstly, for further benchmarking, we prepared\na systematic set of benchmark sequences to evaluate the object tracking capability through long-term and\ncomplex occlusion events, with the evaluation results presented in Table 1. These benchmark sequences, along\nwith the ground truth data, are provided in the Supplementary Information. Subsequent studies can utilize this\nbenchmark set to compare their results with ours.\n4) The ablation study for the important hyper-parameters (\\lambda_large, \\lambda_small, max_age) was performed, and the\ncorresponding subsections have been added to the Methods section. This demonstrates the enhanced object\ntracking capability of the proposed flexible assignment function compared to the baseline using linear\nassignment.\nPoint-by-point responses to Reviewers’ comments:\nResponses to Reviewer #1: (Bold: Reviewers’ comments)\nFirst of all, we thank the Reviewer #1 for his/her careful reading and accurate understanding of our manuscript. All\nthe suggestions/comments given by the Reviewer #1 are based on the reviewer's deep insights and thus are very\nconstructive. Based on his/her reports, we have revised the manuscripts substantially. We believe that the manuscript\nis notably strengthened.\nReviewer#1-Comment#1:\n1. What is the novelty of the proposed Ising/QUBO compared to\nZaech, J. N., Liniger, A., Danelljan, M., Dai, D., & Van Gool, L. (2022). Adiabatic quantum computing for multi-object\ntracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8811-8822).\nThey also proposed QUBO to tackle MOT, and since a QUBO can be turned into an Ising model, their QUBO is\nsolvable by your Ising Machine. Therefore, the manuscript lacks discussion/comparison on the above QUBO model.\nYour paper would be much more interesting if the two models could be compared in terms of qubit/spin complexity\n(the number of variables in the models given Nt and Nd), running time, MOT metrics, and so on. A section is needed\nfor the above paper to be fully discussed. You also need to show the gap that why your model is needed at all and why\nnot using their model on your Ising machine.\nAuthor response:\nWe thank the Reviewer #1 for this constructive suggestion. Following this suggestion, we created a new section\nentitled \"QUBO-based Assignment\" (Lines 321 to 345), which helps clarify \"what is new\" and \"what can be done in\nfuture works.\" Additionally, we revised the Introduction section to imply and emphasize the discussion of the new\nsection, as quoted: \"The QUBO formulation for the flexible assignment is related to, but distinct from, the time-series\nassignment QUBO recently proposed in Ref. [49]. The difference between them is discussed.”\nIn the original version, we cited this reference as Ref. [48] in the Introduction section, as quoted: \"Some literature\n[47-49] has investigated the applicability of quantum mechanics-based Ising machines (quantum annealers) to the\nassignment problems in MOT assuming one-to-one correspondence.\" Note that in the revised version, the suggested\nreference is now Ref. [49] since another new reference has been added according to Reviewer#1-Comment#2.\nAs described in the newly created section, our QUBO and the QUBO in Ref. [49] are fundamentally different in terms\nof target system and the nature of the quadratic term in QUBO. Therefore, they should be distinctly called \"flexible\nassignment QUBO\" and \"time-series assignment QUBO,\" respectively. The flexible assignment QUBO is for online (or\nreal-time) processing, and the quadratic term arises from the necessity to measure the degree of constraint violations\nfor one-to-one correspondence, thereby determining plausible many-to-one assignments. In contrast, the time-series\nassignment QUBO is for offline (or batch) processing, and the quadratic term is needed to correlate the detected\nobjects at different frames (times).\nMoreover, the flexible assignment QUBO and the time-series assignment QUBO might be mutually complementary\nand could be improved by incorporating the quadratic term of each other. In the revised manuscript, this possibility is\nbriefly mentioned and positioned as a topic for future work.\nReviewer#1-Comment#2:\n2. The following paper is closely related to the paper. It should be added in the literature part for interested readers.\nBirdal, T., Golyanik, V., Theobalt, C., & Guibas, L. J. (2021). Quantum permutation synchronization. In Proceedings\nof the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13122-13133).\nAuthor response:\nWe thank the Reviewer #1 for this suggestion. In the revised manuscript, we incorporated this reference as Ref. [48]\nand cited it in the Introduction (Line 55) and \"QUBO-based Assignment\" (Line 324).\nRef. [48] and Ref. [47] describe fundamentally the same idea and provide a theoretical background to the \"time-series\nassignment QUBO\" of Ref. [49]. In the \"QUBO-based Assignment\" subsection, it is cited as follows: \"Zaech et al. have\nrecently introduced a QUBO-based (quadratic) assignment for MOT [49], which is theoretically grounded in Refs.\n[47,48].\"\n==About the order of the responses==\nReviewer#1-Comments #3, #4, #5, #6, #7, #8, and #11:\nAuthor response:\nWe thank the Reviewer #1 for the series of systematic comments (#3, #4, #5, #6, #7, #8, and #11) on the QUBO\nformulation of the flexible assignment problem, which is the core of this work. Based on the principle of 'global to\ndetail,' we will respond to these comments in the following order: (organization) #11, #7; (constraint) #4, #5;\n(hyper-parameter) #3, #6, and #8.\n====\nReviewer#1-Comment#11:\n11. Overall, the paper is not easy to follow (which becomes even worse for unfamiliar readers to Ising/QUBO OR\nMOT). The QUBO formulation should be in the Methods Section. You have a mix of everything in each section\nAuthor response:\nWe appreciate this comment. We also received similar comments from Reviewer#2 (Reviewer#2-Comment#5) and\nReviewer#3 (Reviewer#3-Comment#5).\nThis work spans several fields, including discrete mathematics, application software, quantum/quantum-inspired\ncomputing, and computer engineering. Additionally, the original version required readers to have a certain level of\nknowledge about QUBO and Ising machines.\nTo improve readability and accessibility to a broad readership, we made substantial revisions as follows.\n(i)We revised the Introduction to better describe the problem, objective, proposed method, and experiment.\nAdditionally, we improved the transitions between paragraphs, as highlighted in yellow in the revised manuscript.\n(ii) We created a new paragraph (Lines 88 to 97) at the beginning of the Results section that lists and emphasizes the\nkey features of this work and explains the composition of the following subsections.\n(iii) To provide background information to readers unfamiliar with QUBO and Ising machines, we revised the\nsubsection entitled \"QUBO and Ising Problems\" (Lines 370 to 384) and created a new subsection entitled \"Penalty\nMethod\" (Lines 385 to 401) in the Methods section. The penalty method described there has been widely employed in\nthe fields of Ising machines and QUBO to transform a constrained problem into an unconstrained problem (QUBO),\nwhich is related to the heart of the proposed flexible assignment function.\n(iv) In the revised manuscript, “\\lambda” has been employed to indicate a penalty weight coefficient instead of “c”. From the\nviewpoint of QUBO formulation of assignment problems, a prior work most closely related to this work is Ref. [49], as\nexplained in the response to Reviewer#1-Comment#1. Ref. [49] employed the same penalty method. Using the same\nnotation (\\lambda) would be better for readers to understand and compare.\n(v) The first half of the subsection \"Flexible Assignment\" corresponds to the QUBO formulation of the proposed\nflexible assignment function. This part was reorganized to explain it step-by-step. For details, please see the response\nto Reviewer#1-Comment#7.\n(vi) We added a new paragraph (Lines 271 to 276) at the beginning of the Demonstration subsection to enhance the\npresentation of the subsequent description and revised the Demonstration subsection to improve readability.\n(vii) We have revised the Discussion section to highlight the key features of this work.\nIn this revised version, we’d like to keep the QUBO formulation in the subsection \"Flexible Assignment\" of the\nResults section instead of moving it to the Methods section. This is because the QUBO formulation of the flexible\nassignment problem was proposed in this work, and it is the heart of this work. As our organizational policy, we place\nthe proposed method in the Results section and move the established/pre-existing methods to the Methods section.\nFor the format regulation of Nature Communications, please see the response to Reviewer#2-Comment#3.\nReviewer#1-Comment#7:\n7. Overall, the QUBO needs mathematical proof to show the corresponding minimum-cost state provides the optimal\nsolution to the assignment problem. This has to be in general, not by examples.\nAuthor response:\nWe appreciate this comment. In response, we have revised the \"Flexible Assignment\" subsection to provide a\nmathematical proof demonstrating the equivalence between the target assignment problem (a constrained problem)\nand the QUBO (an unconstrained problem). This revised section spans from Line 107 to Line 148 and includes the\nbound condition for a weight coefficient \\lambda to ensure equivalence.\n(i)In this work, it is crucial to distinguish between an optimization problem with constraints (the target problem) and\nan unconstrained problem corresponding to the target problem before delving into the proposed flexible assignment\nfunction. The necessary background information, including an explanation of the penalty weight coefficient \\lambda, is\nprovided in the \"QUBO and Ising Problems\" and \"Penalty Method\" subsections of the Methods section, referenced at\nLines 109 and 118 in the Main text, respectively.\n(ii) The mathematical proof proceeds step-by-step:\n1. (baseline) A baseline constrained problem when Nt=Nd\n2. (baseline) An unconstrained problem (a QUBO) corresponding to the baseline and the bound condition for \\lambda\n3. (proposed) An assignment problem extended for the case when Nt\\neqNd in both constrained and unconstrained\nforms, along with the bound condition for \\lambda\nReviewer#1-Comment#4:\n4. “(Hpenalty1 + Hpenalty2)=0 if there are no violations”. This should be proved mathematically for “if and only if.”\nAuthor response:\nWe appreciate this comment. This pertains to an unconstrained representation (utilizing a penalty function) of the\nconstrained target problem and has been explained in the response to Reviewer#1-Comment#7 (see above). Also see\nLines 144 to 148 in the Main text.\nReviewer#1-Comment#5:\n5. To make the QUBO explanation clearer, it is better to add a simple example for a few numbers of trackers and\ndetections, and exemplify how the second part of the Eq.2 ensures the one-to-one correspondence (maybe when\nNt>Nd or Nt<Nd).\nAuthor response:\nWe appreciate this comment. We have revised Figs. 2b and 2f to exemplify the similarity matrices. Now, Figs. 2b, 2c,\nand 2d form a set of similarity matrix and two corresponding assignment tables for small \\lambda and large \\lambda when Nt=Nd\nwithout occlusion. Figs. 2f, 2g, and 2h form another set of similarity matrix and two corresponding assignment tables\nfor small \\lambda and large \\lambda when Nt >Nd with occlusion. These two examples are referenced in the main text at Lines 126\nand 151, respectively.\nReviewer#1-Comment#3:\n3. The penalty factor “c” in Eq.2: The authors should provide a bound for this parameter. How to select a value for “c”\nto make sure the constraints defined in Hpenalty1 and Hpenalty2 are satisfied? This should be justified\nmathematically. The state “larger c as 1.0 and smaller c as 0.1” is not sufficient and acceptable.\nAuthor response:\nWe appreciate this comment. We have added a discussion regarding the bound for the penalty weight \\lambda. First, we\nprovide a basic explanation to readers in “Penalty method” subsection” (Lines 385 to 401), describing the equivalence\nbetween constrained and unconstrained problems and the stability of the Ising machine's dynamics. Then, we explain\nthe bound conditions for (large) \\lambda for both the cases when Nt=Nd (Lines 121 to 132) and when Nt >Nd (Lines 149 to\n155). The small \\lambda was determined by the ablation study (see the response to Reviewer#1-Comment#6).\nReviewer#1-Comment#6:\n6. “The table for the larger c more likely satisfies the constraint for one-to-one correspondence, while one for the\nsmaller c more likely tolerates many-to-one assignments (constraint violations):” This state should be proved\nmathematically or justified based on an ablation study.\nAuthor response:\nWe appreciate this comment. The large \\lambda was determined based on the bound condition (see the response to\nReviewer#1-Comment#3), while the small \\lambda was determined by the ablation study.\nIn this revised version, we present the results of the ablation study in the subsection “Ablation study for \\lambdalarge and\n\\lambdasmall” (Lines 459 to 471) along with Figure 5. The results in Fig.5 clearly demonstrate the effect of the proposed\nflexible assignment and the enhanced object tracking capability compared to the baseline using linear assignment.\nReviewer#1-Comment#8:\n8. The two-stage minimization must be justified. Based on what metric means, the authors guarantee that by having\na two-stage minimization, the optimal solution to the initial assignment problem can be obtained/estimated by the\nIsing solver.\nAuthor response:\nWe appreciate this comment. The justification of the two-stage minimization is divided into explanations for\ndetermining the large \\lambda and the small \\lambda, as detailed in the responses to Reviewer#1-Comment#3 and #6. The final\nassignment result (Fig. 2a) can include potential-match states (also see Fig. 2g), which are obtained by arbitrating\nbetween the two assignment tables for the large \\lambda and the small \\lambda as described in Lines 160 to 163. The functionality\nand effectiveness of the two-stage minimization are quantitatively demonstrated in the ablation result (Lines 459 to\n471, Figure 5).\nReviewer#1-Comment#9:\n9. Table 2 needs running time as well\nAuthor response:\nWe appreciate this comment. In response, we created a new subsection entitled “System-wide throughput” (Lines 484\nto 497) in the Methods section, which includes Table 4 listing the measured runtime for the benchmark sequences in\nTable 1 (previously Table 2). This subsection is referred to at Lines 287 to 288 in the Main text.\nReviewer#1-Comment#10:\n10. What are the limitations of the proposed MOT system, and what can be done in future research to improve it?\nAuthor response:\nWe appreciate this comment. We also received similar comments from Reviewer#2 (Reviewer#2-Comment#11).\nThe ablation study for \\lambdasmall (Fig. 5b) clearly shows the upper bound of the enhancement, highlighting the limitations\nof the proposed method. A potential extension to further improve tracking capability beyond this upper bound is\ndescribed as future work in Lines 349 to 357. For a detailed explanation, please refer to our response to\nReviewer#2-Comment#11.\nReviewer#1-Comment#12:\n12. The discussion/conclusion section must be extended.\nAuthor response:\nWe appreciate this comment. We have revised the Discussion (Conclusion) section (Lines 347 to 359) to highlight the\nkey features of this work, including the comparison between the flexible-assignment QUBO (this work) and the\ntime-series QUBO (another recently proposed one), as well as the evaluation results for the new\nsystematically-designed benchmark sequences for occlusion events.\n================\nResponses to Reviewer #2: (Bold: Reviewers’ comments)\nReviewer#2-Comment#1:\nThe paper presents a novel flexible assignment function for existing multi-object tracking methods based on Ising\nmachines. Unlike existing methods it is not limited to linear one-to-one assignments which can be used to detect and\nhandle occlusions to prevent id-switches. While the presented experimental evaluation and the supplementary\nmaterial shows promising results, there are several major concerns regarding the experimental study and the\nmanuscript itself that need to be addressed before the paper is ready for publication.\nAuthor response:\nWe thank the Reviewer #2 for his/her careful reading and deep understanding of our manuscript, and his/her\nsuggestions and comments based on his/her expertise. We also appreciate his/her positive evaluation to the core of\nthis work, i.e., the flexible assignment. Through responding to his/her report, we have improved the manuscript.\nReviewer#2-Comment#2:\nGeneral comments:\n- I suggest to use the term „tracks“ instead of „trackers“ to avoid confusion with tracking algorithms.\nAuthor response:\nThe authors thank the Reviewer #2 for this suggestion and agree with it. In MOT-related papers, several terms such\nas \"track,\" \"tracker,\" or \"tracklet\" have been used to indicate the same concept (e.g., in the baseline paper for this\nwork, Ref. [4], \"tracker\" is used). As the Reviewer #2 suggested, the usage of \"track\" is more appropriate for this work\nto distinguish it from the tracking algorithm/module. Additionally, the usage of \"track\" is often seen in related papers.\nThroughout the revised manuscript, the term \"trackers\" has been replaced with the term \"tracks.\nReviewer#2-Comment#3:\n- Sec. 2 describes the proposed method, so I suggest to use another name than „Results“\nAuthor response:\nThe authors thank the Reviewer #2 for this suggestion based on his/her understanding of the organization of this\nmanuscript. The authors understand this suggestion to be appropriate.\nThe format regulation of Nature Communications (https://www.nature.com/ncomms/submit/article) differs from\nthose of many other academic journals. An article is expected to begin with a section headed Introduction, followed by\nsections headed Results, Discussion, and Methods (in this order). The Results and Methods sections can be divided by\ntopical subheadings, while the Discussion section may not contain subheadings. The authors examined several\nrecently published papers and found that, in most cases, the proposed method is described in a subsection of the\nResults section, possibly because a proposed method can be regarded as part of the research achievement. On the\nother hand, pre-existing methods and other details needed for reproducibility are described in the Methods section.\nTherefore, the authors would like to adhere to the format regulation.\nReviewer#2-Comment#4:\nStrengths:\n- The supplementary videos are helpful in understanding the benefits of the proposed approach.\n- Table 2 convincingly emphasizes that the flexible assignment marginally outperforms the Hungarian matching on 7\nhand-picked movies\nAuthor response:\nWe appreciate this positive feedback. In this revised version, the supplementary videos and Table 1 (previously Table\n2) have been enhanced to include results for newly added benchmark sequences. The new benchmark sequences were\nadded in response to Reviewer#2-Comment#8 and Reviewer#3-Comment#4. For further discussion, please refer to the\nresponses to Reviewer#3-Comment#4.\nReviewer#2-Comment#5:\nWeaknesses:\n- In general, the paper seems to be hard to understand for readers unfamiliar with Ising machines and Muti-object\ntracking and is thus not inappropriate to target a broader audience.\nAuthor response:\nWe appreciate this comment. We also received similar feedback from Reviewer#1 (Reviewer#1-Comment#11) and\nReviewer#3 (Reviewer#3-Comment#5).\nTo improve readability and accessibility for readers unfamiliar with Ising machines and multi-object tracking, we\nhave made the following changes:\n(1) Created a new paragraph at the beginning of the Results section that lists the key features of this work and\nexplains the composition of the following subsections.\n(2) Revised the \"QUBO and Ising Problems\" subsection in the Methods section.\n(3) Added the \"Penalty Method\" subsection in the Methods section.\nAdditionally, we have substantially improved the Introduction, Results, Discussion, and Methods sections. For\nfurther detailed explanation, please see our responses to Reviewer#1-Comment#11.\nReviewer#2-Comment#6:\n- There are many spelling and grammar errors such as missing plurals or articles as well as syntax errors.\nAuthor response:\nWe appreciate this comment. We also received similar feedback from Reviewer#3 (Reviewer#3-Comment#6).\nThroughout the entire paper, we corrected spelling and grammatical errors and revised the sentences to enhance\nreadability.\nReviewer#2-Comment#7:\nMany paragraphs do not connect well to the respective prior paragraph making the overall story of the sections hard\nto follow. For instance, the introduction combines paragraphs on multi-object tracking, Ising machines, embedded\nsystems without transitions that help to guide the reader. Thus I believe that the manuscript needs major revision\nand is not ready for publication yet.\nAuthor response:\nWe appreciate this comment. To make the overall flow of the sections easy to follow, we created a new paragraph\n(Lines 88 to 97) at the beginning of the Results section that explains the composition of the following subsections and\nserves as a guide. As the Reviewer #2 pointed out, the Introduction section is crucial for helping the reader foresee\nand grasp the structure of the paper. We carefully revised the transitions between paragraphs, as highlighted in\nyellow in the revised manuscript.\nReviewer#2-Comment#8:\n- The evaluation is very limited.\nAuthor response:\nWe appreciate this comment, which is related to Reviewer#3-Comment#4. In this revised version, we have enhanced\nthe evaluation in the following three ways.\n(1) Responding to Reviewer#3-Comment#4, we have newly prepared a systematic set of benchmark sequences to\nevaluate the object tracking capability through long-term and complex occlusion events.\n(2) Responding to Reviewer#2-Comment#11, we have added an ablation study for \\lambdalarge and \\lambdasmall.\n(3) Responding to Reviewer#2-Comment#10, we have added an ablation study for max_age.\nWe believe the revised manuscript includes sufficient evaluation results to justify the significance of the proposed\nMOT system with an embedded Ising machine.\nReviewer#2-Comment#9:\n- c is set to 0.1 and 1.0 . A transparent discussion to show how sensitive the approach is to this hyperparameter would\nbe helpful.\nAuthor response:\nThis comment is closely related to Reviewer#2-Comment#11. Let us address this comment in our response to\nReviewer#2-Comment#11 (see below). Note that in the revised manuscript, the notation “c” has been replaced with “\\lambda”.\nSee our response to Reviewer#1-Comment#11.\nReviewer#2-Comment#10:\n- The authors argue that their method based on „potential matches“ is better than extending the max-age of existing\nMOT methods. It would be good to also show this qualitatively.\nAuthor response:\nWe appreciate this comment. In the revised manuscript, we have added a new subsection titled \"Ablation Study for\nmax_age \" (Lines 472 to 483) and included the corresponding experimental results (Figure 6) in the Methods section.\nThis subsection, along with the relevant Main text (Lines 230 to 236), quantitatively and qualitatively demonstrates\nthe superiority of the proposed method over the existing baseline method with an increased max_age.\nReviewer#2-Comment#11:\n- An ablation for the parameters of the Ising machine would be helpful in understanding the proposed methods’\nlimitations\nAuthor response:\nWe appreciate this comment, which is closely related to Reviewer#2-Comment#9. Additionally, we received a series of\nrelated comments from Reviewer#1 (Reviewer#1-Comments #3, #4, #5, #6, #7, #8, and #11).\nThe penalty weight \\lambda is an important hyper-parameter and closely related to the proposed flexible assignment\nfunction, which is the core of this work. Depending on \\lambda, the Ising machine outputs different solutions (assignment\ntables) for the same input (similarity matrix). We utilize two values of \\lambda (\\lambdalarge and \\lambdasmall) to realize the proposed\nflexible assignment function.\nIn this revised version, we present the results of an ablation study in the subsection “Ablation study for \\lambdalarge and\n\\lambdasmall” (Lines 459 to 471), along with Figure 5. The results in Fig. 5b clearly demonstrate the enhanced object tracking\ncapability compared to the baseline using linear assignment. Furthermore, as stated in Lines 468 to 471, the results\nin Fig.5b clearly show the upper bound of the enhancement (the proposed methods’ limitations) when decreasing \\lambdasmall\n(the effect of the flexible assignment is strengthened as \\lambdasmall decreases). This is due to a side effect arising from a\ntoo-small penalty weight, as mentioned in Lines 470 to 471.\nTo further improve tracking capability beyond this upper bound, it would be effective to refine the similarity using\nadvanced machine learning technology and/or introduce quadratic elements to the similarity matrix (conditional\nlikelihood or tradeoff relationship). This potential extension is discussed as future work in Lines 360 to 365.\n================\nResponses to Reviewer #3: (Bold: Reviewers’ comments)\nWe thank the Reviewer #3 for his/her careful reading and helpful comments on this manuscript. The Reviewer #3\nspecifically listed the technical issues to be addressed, including an important one regarding further benchmarking.\nBy responding to all the suggestions and comments from the Reviewer #3, we have substantially improved the\nmanuscript.\nReviewer#3-Comment#1:\nThis manuscript aims to present an in-vehicle multiple-object tracking system with an embeddable Ising machine\nbased on a quantum-inspired algorithm.\nHowever, there are many issues with this manuscript.\n1) The contributions are unclear.\nAuthor response:\nWe appreciate this comment. In response, we have revised the Introduction and Discussion (Conclusion) sections to\nclearly describe the problem, objective, proposed method, and contributions.\nThe main contributions are:\n(1) Flexible assignment function: The proposal of a new flexible assignment function to enhance object tracking\ncapability through long-term and complex occlusion events, which is enabled by emerging embeddable Ising\nmachines.\n(2) QUBO formulation: The QUBO formulation of the proposed flexible assignment.\n(3) Real-time operation: The demonstration of real-time operation and enhanced assignment functionality under\nconstraints of size, power, and cost.\n(4) Comparison with time-series assignment QUBO: The proposed QUBO for flexible assignment is compared with\nanother recently proposed QUBO for time-series assignment.\nWith these contributions, this work paves the way for enhancing the functionality of vehicle-mountable real-time\nsystems with embeddable Ising machines.\nReviewer#3-Comment#2:\n2) The research issues and the response methods are unclear. The proposed method is weak.\nAuthor response:\nWe appreciate this comment. In response, we have revised the Introduction section.\nThe research issues are:\n(1) To propose and demonstrate the possibility of enhancing the functionality of vehicle-mountable real-time systems\nwith embeddable Ising machines.\n(2) To propose a new practical function enabled by embeddable Ising machines.\n(3) To demonstrate the enhanced functionality under constraints of size, power, and cost for vehicle-mountable\nsystems.\nThe proposed method has the following strong points:\n(1) The embeddable Ising machine utilized in this work, based on the recently discovered quantum-inspired\nalgorithm known as simulated bifurcation, has demonstrated high-performance and energy-efficient, making it\nsuitable for real-time vehicle-mountable systems.\n(2) The enhancement of functionality in a real-time vehicle-mountable systems through NP-hard combinatorial\noptimization has been proposed and demonstrated with a concrete example: the flexible assignment function.\nAs stated in Lines 357 to 359, the concept of a vehicle-mountable computing platform with embedded Ising machines\ncould be applicable to various tasks other than MOT, such as simultaneous localization and mapping (SLAM),\nscheduling, or routing.\nReviewer#3-Comment#3:\n3) The experiments that were conducted did not show remarkable results. Current results are weak and can not\nsupport the proposed system and the claimed goal.\nAuthor response:\nWe appreciate this comment. In response to Reviewer#3-Comment#4 and other comments from Reviewer#1 and\nReviewer#2, the Demonstration section has been substantially strengthened. Firstly, for further benchmarking, we\nprepared a systematic set of benchmark sequences to evaluate the object tracking capability through long-term and\ncomplex occlusion events, with the evaluation results presented in Table 1. The ablation study for the important\nhyper-parameters (\\lambdalarge, \\lambdasmall, max_age) was performed, and the corresponding subsections have been added to the\nMethods section. These results clearly demonstrate the enhanced object tracking capability of the proposed flexible\nassignment function compared to the baseline using linear assignment.\nReviewer#3-Comment#4:\n4) The manuscript claims an in-vehicle multiple-object tracking system, however, the proposed system has not been\ndemonstrated on well-known driving datasets such as nuScenes. Furthermore, the proposed system claims a\nreal-time in-vehicle MOT system; however, the system has not been demonstrated in a vehicle system and driving\nscenarios. Instead, an embedded Ising machine was used to test the system, which can not demonstrate the\ncontributions of the proposed in-vehicle system.\nAuthor response:\nWe appreciate this comment. The Reviewer #3 highlighted two issues: further demonstration/benchmarking and the\nterminology of “in-vehicle.” Let's first address the latter.\nRegarding the terminology of “in-vehicle”, this work has demonstrated a vehicle-mountable system with an embedded\nIsing machine, aimed at enhancing future in-vehicle systems. However, as the Reviewer #3 pointed out, we have not\ndemonstrated an in-vehicle system itself. In the revised manuscript, we strictly distinguish between the terms\n“in-vehicle” and “vehicle-mountable”. The former is used in the context of future directions, while the latter is used in\nthe context of actual demonstrations.\nFor further demonstration and benchmarking, we have newly prepared a systematic set of benchmark sequences\ndesigned to evaluate tracking capability through long-term and complex occlusion events. These benchmark\nsequences, along with the ground truth data, are provided in the Supplementary information. Subsequent studies can\nutilize this benchmark set to evaluate and compare their results with those presented in this paper. The benchmark\nsequences are explained in the subsection of the Method section (Lines 453 to 458). The evaluation results of the\nbenchmark sequence are presented in Table 1 and discussed in Lines 302 to 320.\nThis work proposes and focuses on an enhanced matching module (assignor in Fig.1) within tracking systems, rather\nthan a unified in-vehicle cognitive system. It is crucial to compare the proposed matching module (flexible\nassignment) with the baseline (linear assignment) in terms of tracking capability using practical and academically\ndesigned benchmark sequences.\nMany recent MOT papers [4-7,9-11,49] utilize standard benchmark sequences known as MOT challenges [62,63],\nwhich are based on 2D-camera videos and provided along with ground truth data including annotated objects with\nIDs and 2D-boundary boxes. The ground truth data is necessary to calculate an MOT evaluation metric such as HOTA\nusing a widely used evaluation tool known as TrackEval [65]. While MOT challenges are based on actual driving\nscenes, they are not specifically designed to evaluate various occlusion scenes, where the occurrence of occlusion\nevents is relatively moderate.\nWe agree with Reviewer #3’s comment regarding further benchmarking. To clarify the enhancements made by this\nwork compared to the baseline, we believe it is important to assess our proposed matching module using academically\ndesigned benchmarks for tracking through complex occlusion events. The results from these academically designed\nbenchmarks complement those from the standard benchmarks based on actual driving scenes (MOT challenges).\nWe thank the Reviewer #3 for suggesting the driving datasets known as nuScenes, which we have added as Ref. [71].\nThe nuScenes dataset is intended for unified in-vehicle cognitive systems, based on multimodal sensors (6 cameras, 5\nradars, and 1 lidar, all with a full 360-degree field of view) and provided along with annotated objects with\n3D-boundary boxes. The data format provided is not applicable to the evaluation scheme of MOT challenges [62,63]\nand TrackEval [65]. It would be possible to produce a similarity matrix by unifying information from multimodal\nsensers (with an advanced version of detector and associator in Fig. 1) and then determine the assignment result by\nthe assignor proposed in this work. However, defining a similarity matrix based on multimodal sensers is beyond the\nscope of this work. Therefore, we position that as future work, as stated in Lines 360 to 362. \nReviewer#3-Comment#5:\n5) The structure of this manuscript is confused. The research goal, proposed solution, experiments, and results must\nbe designed consistently.\nAuthor response:\nWe appreciate this comment. We also received similar comments from Reviewer#1 (Reviewer#1-Comment#11), and\nReviewer#2 (Reviewer#2-Comment#5).\nFirstly, we revised the Introduction to better describe the problem, objective, proposed method, and experiment.\nSecondly, we substantially revised the Results, Discussion, and Methods sections to improve the consistency\nthroughout the manuscript. For further detailed explanation, please see the responses to Reviewer#1-Comment#11.\nReviewer#3-Comment#6:\n6) The writing in this manuscript needs to be significantly improved.\nAuthor response:\nWe appreciate this comment. We received similar feedback from Reviewer#2 (Reviewer#2-Comment#6).\nThroughout the entire paper, we corrected spelling and grammatical errors and revised the sentences to enhance\nreadability."}, {"round": 2, "reviewer_comments": {"reviewer_1": "I would like to thank the authors for their detailed and thoughtful responses to the review comments. They have\naddressed all concerns raised during the review process comprehensively. As far as I am concerned, the paper is\nmathematically sound and methodologically robust. The results are clearly presented, and the comparisons are\nconvincing. I therefore recommend acceptance of the paper.", "reviewer_3": "Thanks for the responses from the authors. However, the new version still needs to be improved. Here, I would like to\nupdate my comments based on the previous comments. Hope the authors can improve this paper by following the\nupdated comments. Please focus on contributing high-quality work rather than responding to the reviewers with a\nletter.\n1) Please provide hardcore contributions and clear writing. The current contributions are weak and unclear.\n2) Please design the paired research questions/goals and the response methods. The current research questions/goals and\nthe response methods are weak and unclear.\n3) Please provide the solid results to demonstrate the claimed research goals in this work. The experiments that were\nconducted did not demonstrate remarkable results. The current results are weak and can not demonstrate the proposed\nsystem and the claimed goals in this work.\n4) The manuscript claims an in-vehicle multiple-object tracking system, however, the proposed system has not been\ndemonstrated on well-known driving datasets such as nuScenes. Furthermore, the proposed system claims a real-time in-\nvehicle MOT system; however, the system has not been demonstrated in a vehicle system and driving scenarios. Instead,\nan embedded Ising machine was used to test the system, which can not demonstrate the contributions of the proposed in-\nvehicle system.\n5) Please provide a clear structure. The current structure of this manuscript is confused. Importantly, the research goal,\nproposed solution, experiments, and results must be designed consistently.\n6) Please note that the writing in this manuscript needs to be significantly improved."}, "author_response": "Response to Reviewers:\nWe appreciate the time and effort the reviewers devoted to providing feedback on this manuscript. We have addressed\nall the comments from the reviewers, which has further improved the manuscript.\nPoint-by-point responses to Reviewers’ comments:\nResponses to Reviewer #1: (Bold: Reviewers’ comments)\nReviewer#1-Comment#1:\nI would like to thank the authors for their detailed and thoughtful responses to the review comments. They have\naddressed all concerns raised during the review process comprehensively. As far as I am concerned, the paper is\nmathematically sound and methodologically robust. The results are clearly presented, and the comparisons are\nconvincing. I therefore recommend acceptance of the paper.\nAuthor response:\nWe sincerely thank Reviewer #1 for the constructive feedback provided throughout the review process. We greatly\nappreciate their support of our manuscript.\n================\nResponses to Reviewer #3: (Bold: Reviewers’ comments)\nThanks for the responses from the authors. However, the new version still needs to be improved. Here, I would like to\nupdate my comments based on the previous comments. Hope the authors can improve this paper by following the\nupdated comments. Please focus on contributing high-quality work rather than responding to the reviewers with a\nletter.\nWe thank Reviewer #3 for their thoughtful feedback. Based on the updated comments, we have revised the\nmanuscript to clarify its positioning and goals, and to improve its overall quality. We have attached a manuscript file\nhighlighting the important revisions made in response to Reviewer #3’s comments.\nReviewer#3-Comment#1:\n1) Please provide hardcore contributions and clear writing. The current contributions are weak and unclear.\nReviewer#3-Comment#2:\n2) Please design the paired research questions/goals and the response methods. The current research questions/goals\nand the response methods are weak and unclear.\nReviewer#3-Comment#3:\n3) Please provide the solid results to demonstrate the claimed research goals in this work. The experiments that were\nconducted did not demonstrate remarkable results. The current results are weak and can not demonstrate the\nproposed system and the claimed goals in this work.\nAuthor response:\nWe thank Reviewer #3 for Comments #1 through #3. As these comments are closely related, we address them\ntogether.\nIn response to Comments #1 and #2, we have revised the goal/contribution statement (Lines 63–66) as follows:\n“To demonstrate the potential and feasibility of enhancing vehicle-mountable control systems using emerging Ising\nmachines, we propose and implement a real-time MOT system featuring an enhanced temporal association\nmechanism, called the flexible assignment function, which is based on NP-hard combinatorial optimization and\nenabled by an embedded Ising machine.”\nThis statement outlines both the research goal (“to demonstrate...”) and the proposed method (“we propose and\nimplement...”), thereby addressing the request for clearer research questions and response methods, as specifically\nraised in Comment #2.\nRegarding Comment #3, we have organized the demonstration of our contributions into five key points:\n(1) Introduction of the concept of vehicle-mountable systems with embeddable Ising machines, along with a specific\nand practical use case (MOT)\n(2) Proposal of the flexible assignment QUBO (how to leverage Ising machines)\n(3) Evaluation of feasibility under size, power, and cost constraints\n(4) Demonstration of real-time processing capability\n(5) Demonstration of enhanced MOT functionality compared to the baseline using conventional linear assignment.\nThese contributions are supported by solid experimental results, summarized as follows:\nResult Corresponding point(s) Result Corresponding point(s)\nFigure 1 (1) Table 1 (5)\nFigure 2 (2) Table 2 (3)\nFigure 3 (3) Table 3 (5)\nFigure 4 (4)(5) Table 4 (4)\nFigure 5 (5) Supplementary Fig. 1 (5)\nThese clarifications are reflected in the updated Introduction and Discussion sections.\nReviewer#3-Comment#4:\n4) The manuscript claims an in-vehicle multiple-object tracking system, however, the proposed system has not been\ndemonstrated on well-known driving datasets such as nuScenes. Furthermore, the proposed system claims a\nreal-time in-vehicle MOT system; however, the system has not been demonstrated in a vehicle system and driving\nscenarios. Instead, an embedded Ising machine was used to test the system, which can not demonstrate the\ncontributions of the proposed in-vehicle system.\nAuthor response:\nWe thank Reviewer #3 for Comment #4. We have considered the background of this comment and revised the\nmanuscript as follows.\nAs clarified in our response to Comments #1 through #3, our research goal is “to demonstrate the potential and\nfeasibility of enhancing vehicle-mountable control systems using emerging Ising machines.” Hence, the embedded\nIsing machine is the main topic of this manuscript. As described in Lines 39–62 of the revised Introduction, Ising\nmachines are an emerging technology. The research field of Ising machines was initiated by the introduction of\nD-Wave’s quantum annealer in 2011, and our research goal lies at the cutting edge of this field.\nTo demonstrate our research goal, we provide a concrete and practical example: the proposed vehicle-mountable MOT\nsystem enhanced by an embedded Ising machine. To show the potential and significance of the embedded Ising\nmachine, the direct comparison between the proposed system with the flexible assignment QUBO (NP-hard quadratic\nassignment) and the baseline system using conventional linear assignment is critical. We do not intend to claim a\nstate-of-the-art MOT system in terms of comprehensive evaluation or a demonstration of a real in-vehicle system.\nInstead, we have demonstrated an enhanced MOT functionality enabled by the embedded Ising machine: a flexible\nassignment function capable of tracking through long-term and complex occlusion events. As discussed in the\nDiscussion section as future work, the demonstrated embeddable Ising machine can be combined with various\nbaseline MOT systems and could be applied to systems other than MOT, such as simultaneous localization and\nmapping (SLAM), scheduling, or path planning. Additionally, embeddable Ising machines are not limited to mobile\nvehicles but are applicable to various mobile systems like robots, as described in the revised Introduction section.\nIn the revised manuscript, we replaced the term “in-vehicle” with “vehicle-mountable” because it more accurately\nrepresents our research goal and the demonstrated system (see Fig. 3 and Supplementary Information 2).\nIn this context, we do not believe that a demonstration using the nuScenes dataset (cited as Ref. [71]) is necessary.\nAdditionally, in the first rebuttal letter, we provided the following explanation regarding nuScenes:\n“The nuScenes dataset is intended for unified in-vehicle cognitive systems, based on multimodal sensors (6 cameras, 5\nradars, and 1 lidar, all with a full 360-degree field of view) and provided along with annotated objects with\n3D-boundary boxes. The data format provided is not applicable to the evaluation scheme of MOT challenges [62,63]\nand TrackEval [65]. It would be possible to produce a similarity matrix by unifying information from multimodal\nsensors (with an advanced version of detector and associator in Fig. 1) and then determine the assignment result by\nthe assignor proposed in this work. However, defining a similarity matrix based on multimodal sensors is beyond the\nscope of this work. Therefore, we position that as future work, as stated in Lines 360 to 362 (Lines 367 to 369 in the\nrevised version).”\nReviewer#3-Comment#5:\n5) Please provide a clear structure. The current structure of this manuscript is confused. Importantly, the research\ngoal, proposed solution, experiments, and results must be designed consistently.\nAuthor response:\nWe thank Reviewer #3 for Comment #5. In response to Comments #1 through #3, we have revised the statement of\nour research goal and contributions (Lines 63–66). We believe that the updated goal statement, along with the\nimproved Introduction, enhances the clarity of the manuscript’s overall structure.\nAs noted in our response to Comment #3, we have organized the presentation of our contributions into the following\nfive key points:\n(1) Introduction of the concept of vehicle-mountable systems with embeddable Ising machines, along with a specific\nand practical use case (MOT)\n(2) Proposal of the flexible assignment QUBO (how to leverage Ising machines)\n(3) Evaluation of feasibility under size, power, and cost constraints\n(4) Demonstration of real-time processing capability\n(5) Demonstration of enhanced MOT functionality compared to the baseline using conventional linear assignment.\nThe manuscript has been structured to present these five points in the above order. We believe this structure is\nstraightforward and consistent with the revised goal statement.\nFurthermore, to improve readability, we have made the following structural revisions:\n(i) The Introduction has been revised to more clearly describe the problem setting, research objective, proposed\nmethod, and experimental approach.\n(ii) A new paragraph has been added at the beginning of the Results section to summarize the key features of this\nwork and to explain the organization of the subsequent subsections.\n(iii) A new introductory paragraph has been added to the Demonstration subsection to improve the flow and clarity of\nthe subsequent content.\n(iv) The Discussion section has been revised to better highlight the key contributions and implications of this work.\nReviewer#3-Comment#6:\n6) Please note that the writing in this manuscript needs to be significantly improved.\nAuthor response:\nWe thank Reviewer #3 for Comment #6. We have improved the writing throughout the manuscript based on this\nfeedback. Specifically, we have implemented the following changes:\n-All Supplementary Information files (1 through 14) are now explicitly cited in the main text to ensure clarity and\ntraceability.\n-All acronyms appearing in figures have been defined in the corresponding captions.\n-All figure legends have been revised to begin with concise titles that summarize the content.\n-All figures have been updated to improve legibility, including adjustments to font size, color schemes, and labeling.\n-We have thoroughly reviewed the entire manuscript to correct spelling and grammatical errors."}]

Decision letter