Patent ID: 12243299

DESCRIPTION OF EXAMPLE EMBODIMENTS

Next, an example embodiment of an object tracking apparatus, an object tracking method and a recording medium will be described with reference to the drawings. In the below described description, an object tracking apparatus1to which the example embodiment of the object tracking apparatus, the object tracking method and the recording medium is applied will be described. The object tracking apparatus1performs an object tracking operation for tracking an object included in each image IM when a plurality of images IM corresponding to time-series data are inputted thereto. The object tracking operation includes an object detection operation for detecting an object O included in the image IM, for example. The object tracking operation includes an object matching operation for tracking a position of an object Ot-τ, which has been included in an image IMt-τ, in an image IMtby performing a matching between at least one object Ot-τincluded in the image IMt-τobtained (for example, captured) at a time t-τ (note that τ indicates a reference cycle) and at least one object Otincluded in the image IMtobtained at a time t, for example. The object tracking operation includes a refinement operation for correcting a detected result of the object Otincluded in the image IMt(namely, a result of the object detection operation) by using a result of the object matching operation.

Note that an expression “X and/or Y” is used as an expression including both of an expression “X and Y” and an expression “X or Y” in the below described description.

(1) Configuration of Object Tracking Apparatus1

Firstly, with reference toFIG.1andFIG.2, a configuration of the object tracking apparatus1in the present example embodiment will be described.FIG.1is a block diagram that illustrates the configuration of the object tracking apparatus1in the present example embodiment.FIG.2is a block diagram that illustrates a configuration of logical functional blocks implemented in the object tracking apparatus1for performing the object matching operation and the refinement operation.

As illustrated inFIG.1, the object tracking apparatus1includes an arithmetic apparatus2and a storage apparatus3. Furthermore, the object tracking apparatus1may include an input apparatus4and an output apparatus5. However, the object tracking apparatus1may not include at least one of the input apparatus4and the output apparatus5. The arithmetic apparatus2, the storage apparatus3, the input apparatus4and the output apparatus5are interconnected through a data bus6.

The arithmetic apparatus2includes at least one of a CPU (Central Processing Unit) and GPU (Graphical Processing Unit, for example. The arithmetic apparatus2reads a computer program. For example, the arithmetic apparatus2may read a computer program stored in the storage apparatus3. For example, the arithmetic apparatus2may read a computer program stored in a computer-readable non-transitory recording medium, by using a not-illustrated recording medium reading apparatus. The arithmetic apparatus2may obtain (namely, download or read) a computer program from a not-illustrated apparatus placed outside the object tracking apparatus1through a not-illustrated communication apparatus. The arithmetic apparatus2executes the read computer program. As a result, a logical functional block for performing an operation (specifically, the above described object tracking operation) that should be performed by the object tracking apparatus1is implemented in the arithmetic apparatus2. Namely, the arithmetic apparatus2is configured to serve as a controller for implementing the logical block for performing the object tracking operation.

FIG.1illustrates one example of the logical functional block that is implemented in the arithmetic apparatus2for performing the object tracking operation. As illustrated inFIG.1, an object detection unit21, an object matching unit22and a refinement unit23are implemented in the arithmetic apparatus2as the logical functional blocks. The object detection unit21performs the object detection operation. The object matching unit22performs the object matching operation. In order to perform the object matching operation, as illustrated inFIG.2, the object matching unit22includes a feature map conversion unit221, a feature vector conversion unit222, a feature map conversion unit223, a feature vector conversion unit224and a matrix calculation unit225. The refinement unit23performs the refinement operation. In order to perform the refinement operation, as illustrated inFIG.2, the refinement unit23includes a matrix calculation unit231, a feature vector conversion unit232, a feature map conversion unit233, a residual processing unit234and a feature map conversion unit235. Note that the object detection operation, the object matching operation and the refinement operation will be described later in detail.

The storage apparatus3is configured to store desired data. For example, the storage apparatus3may temporarily store the computer program that is executed by the arithmetic apparatus2. The storage apparatus3may temporarily store data temporarily used by the arithmetic apparatus2when the arithmetic apparatus2executes the computer program. The storage apparatus3may store data stored for a long term by the object tracking apparatus1. Especially in the present example embodiment, the storage apparatus3may store an image DB (DataBase)31for storing the plurality of images IM that are captured by a not-illustrated camera and that are the time-series data. Furthermore, the storage apparatus3may store an object detection DB32for storing an object detection information indicating the result of the object detection operation (namely, an information related to the detected result of the object O included in the image IM). Furthermore, the storage apparatus3may store an object matching DB33for storing an object matching information indicating the result of the object matching operation (namely, an information related to a matching result of the object Ot-τincluded in the image IMt-τand the object Otincluded in the image IMt-τ, and typically an information indicating a correspondence between the object Ot-τand the object Ot). Note that the storage apparatus3may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk apparatus, a magneto-optical disc, a SSD (Solid State Drive) and a disk array apparatus. Namely, the storage apparatus3may include a non-transitory recording medium.

An input apparatus4is an apparatus that is configured to receive an input of an information from an outside of the object tracking apparatus1to the object tracking apparatus1. In the present example embodiment, the plurality of images IM that are captured by the not-illustrated camera and that are the time-series data are inputted to the input apparatus4. The plurality of images IM inputted to the input apparatus4are stored in the image DB31stored in the storage apparatus3.

The output apparatus5is an apparatus that is configured to output an information to an outside of the object tracking apparatus1. For example, the output apparatus5may output an information related to a result of the object tracking operation. For example, the output apparatus5may output, as an image, the information related to the result of the object tracking operation. In this case, the output apparatus5may include a display apparatus that is configured to display the image. For example, the output apparatus5may output, as data, the information related to the result of the object tracking operation. In this case, the output apparatus5may include a data output apparatus that is configured to output the data.

(2) Object Tracking Operation Performed by Object Tracking Apparatus1

Next, the object tracking operation performed by the object tracking apparatus1will be described. As described above, the object tracking operation includes the object detection operation, the object matching operation and the refinement operation. Therefore, in the below described description, the object detection operation, the object matching operation and the refinement operation will be described in sequence.

(2-1) Object Detection Operation

Firstly, the object detection operation performed by the object detection unit21will be described. The object detection unit21reads the image IM stored in the image DB31and performs the object detection operation on the read image IM. The object detection unit21may detect the object O included in the image IM by using an existing method of detecting an object included in an image. However, it is preferable that the object detection unit21perform the object detection operation by using a method that is capable of obtaining an information (hereinafter, it is referred to as an “object position information PI”) related to a position of the object O in the image IM by detecting the object O included in the image IM. The object position information PI obtained by the object detection unit21is stored in the object detection DB32as the object detection information indicating the result of the object detection operation by the object detection unit21.

In the present example embodiment, an example in which the object detection unit21detects the object O by using a method disclosed in the above described non-Patent Literature 3 (what we call a CenterNet). In this case, the object detection unit21generates, as the object position information PI, a heat map (what we call a score map) indicating a center point (a Key Point) KP of the object O in the image IM. More specifically, the object detection unit21generates the heat map indicating the center point KP of the object O in the image IM for each class of the object O. Thus, an information indicating the center point KP of the object O in the image IM is a map information having a size of H×W×K, wherein H is the H is the number of pixel of the image IM along a vertical direction, W is the number of pixel of the image IM along a horizontal direction and K is the number of class. Note that the heat map indicating the center point KP may be referred to as a position map, because it is a map related to the position.

When the object detection unit21detects the object O by using the method disclosed in the non-Patent Literature 3, the object detection unit21may generate, as the object position information PI, an information that indicates a size of a detection frame (a Bounding Box) BB of the object O as a score map. The information indicating the size of the detection frame BB of the object O may be regarded to be substantially an information indicating a size of the object O. The information indicating the size of the detection frame BB of the object O is a map information having a size of H×W×2, for example. Note that the map information indicating the size of the detection frame BB of the object O may be referred to as the position map too, because it is a map related to the position.

When the object detection unit21detect the object O by using the method disclosed in the non-Patent Literature 3, the object detection unit21may generate, as the object position information PI, an information that indicates a correction amount (a Local Offset) of the detection frame of the object O as the score map. The information indicating the correction amount of the detection frame BB of the object O is a map information having a size of H×W×2. Note that the map information indicating the correction amount of the detection frame BB may be referred to as the position map too, because it is a map related to the position.

FIG.3is a planar view that conceptually illustrates the object position information PI detected by the object detection operation.FIG.3illustrates an example in which four objects O (specifically, an object O #1, an object O #2, an object O #3and an object O #4) are included in the image IM. In this case, the object detection unit21generates, as the object position information PI, the information indicating the center point KP of each of the four objects O, the information indicating the size of the detection frame BB of each of the four objects O and the information indicating the correction amount of the detection frame BB of each of the four objects O.

The object detection unit21may perform the object detection operation by using a calculation model that is configured to output the object position information PI when the image IM is inputted thereto. One example of the calculation model is a calculation model using a Neural Network (for example, a CNN: Convolutional Neural Network). In this case, a learning of the calculation model may be performed by using learning data, as described later. Namely, a parameter of the calculation model may be optimized so as to output the proper object position information PI.

However, the object detection unit21may perform the object detection operation by using another method that is different from the method (what we call the CenterNet) disclosed in the non-Patent Literature 3. A method that is disclosed in the non-Patent Literature 4 and that is referred to as a “Faster R-CNN” and a method that is disclosed in the non-Patent Literature 5 and that is referred to as a “SSD” are examples of another method.

(2-2) Object Matching Operation

Next, the object matching operation performed by the object matching unit22will be described. The object matching unit22reads the object position information PI stored in the object detection DB32and performs the object matching operation by using the read object position information PI. In the below described description, with reference toFIG.4, the object matching operation for performing the matching between the object Ot-τincluded in the image IMt-τcaptured at the time t-τ and the object Otincluded in the image IMtcaptured at the time t will be described.FIG.4is a flowchart that illustrates a flow of the object matching operation.

As illustrated inFIG.4, the object matching unit22generate a feature vector CV from the object position information PI (a step S221to a step S226). In the present example embodiment, the object matching unit22generates a feature map CM from the object position information PI, and then generates the feature vector CV form the feature map CM, in order to generate the feature vector CV from the object position information. The feature map CM is a feature map that indicates a feature of the object position information PI for each channel. Namely, when the number of the channel is C, the object matching unit22generates the feature map CM having a size of H×W×C from the object position information PI that is the map information having a size of H×W×(K+2+2)=H×W×(K+4) (K is the number of the class of the object O as described above, typically, the number of the class registered in the object detection DB32or the object matching DB33), and then, generates the feature vector CV having a size of HW×C from the feature map CM having the size of H×W×C. However, the object matching unit22may directly generate the feature vector CV from the object position information PI without generating the feature map CM.

Specifically, the feature map conversion unit221that is one specific example of a “first generation unit” obtains, from the object detection DB32, an object position information PIt-τrelated to the object Ot-τincluded in the image IMt-τcaptured at the time t-τ (the step S221). Note that the object position information PIt-τis one specific example of a first position information. For example, when the four objects Ot-τare included in the image IMt-τas illustrated inFIG.3, the feature map conversion unit221obtains, from the object detection DB32, an object position information PIt-τrelated to the four objects Ot-τ. Then, the feature map conversion unit221generates a feature map CMt-τfrom the object position information PIt-τobtained at the step S221(the step S222). The feature map conversion unit221generates the feature map CMt-τhaving the size of H×W×C from the object position information PIt-τthat is the map information having the size of H×W×(K+4).

For example, the feature map conversion unit221may generate the feature map CMt-τby using a calculation model that is configured to output the feature map CM when the object position information PI is inputted thereto. One example of the calculation model is a calculation model using a Neural Network (for example, a CNN: Convolutional Neural Network). In this case, a learning of the calculation model may be performed by using learning data, as described later. Namely, a parameter of the calculation model may be optimized so as to output the proper feature map CM (especially, the feature map CM that is suitable for generating an affinity matrix AM described later).

Then, the feature vector conversion unit222that is one specific example of the “first generation unit” generates a feature vector CVt-τfrom the feature map CMt-τgenerated at the step S222(the step S223). The feature vector conversion unit222generates the feature vector CVt-τhaving the size of HW×C from the feature map CMt-τhaving the size of H×W×C. Note that the feature vector CVt-τis one specific example of a first feature vector.

In parallel with, after or before the processing from the step S221to the step S223, the feature map conversion unit223that is one specific example of the “first generation unit” obtains, from the object detection DB32, an object position information PItrelated to the object Otincluded in the image IMtcaptured at the time t (the step S224). Note that the object position information PItis one specific example of a second position information. Then, the feature map conversion unit223generates a feature map CMtfrom the object position information PItobtained at the step S224(the step S225). The feature map conversion unit221generates the feature map CMthaving the size of H×W×C from the object position information PItthat is the map information having the size of H×W×(K+4). Note that a detail of the processing from the step S224to the step S225may be same as a detail of the processing from the step S221to the step S222described above. Thus the feature map conversion unit223may generate the feature map CMtby using the calculation model that is configured to output the feature map CM when the object position information PI is inputted thereto, as with the feature map conversion unit221.

Then, the feature vector conversion unit224that is one specific example of the “first generation unit” generates a feature vector CVtfrom the feature map CMtgenerated at the step S225(the step S226). The feature vector conversion unit224generates the feature vector CVthaving the size of HW×C from the feature map CMthaving the size of H×W×C. Note that a detail of the processing at the step S226may be same as a detail of the processing at the step S223described above. Note that the feature vector CVtis one specific example of a second feature vector.

Then, the matrix calculation unit225that is one specific example of a “second generation unit” generates the affinity matrix AM by using the feature vector CVt-τgenerated at the step S223and the feature vector CVtgenerated at the step S226(a step S227). Specifically, the matrix calculation unit225generates, as the affinity matrix AM, an information that is obtained by a calculation processing using the feature vector CVt-τand the feature vector CVt.

For example, the matrix calculation unit225may generate, as the affinity matrix AM, an information that is obtained by a calculation processing for calculating a matrix product (a matrix multiplication) of the feature vector CVt-τand the feature vector CVt(the matrix product of the feature vector CVt-τand the feature vector CVt.). The matrix product here may be a tensor product (a direct product). As one example, the matrix product may be a Kronecker product. In this case, a size of the affinity matrix is HW×HW.

As a result, as illustrated inFIG.5that is planar view conceptually illustrating a relationship between the feature vectors CVt-τand CVtand the affinity matrix AM, the affinity matrix AM is an information indicating a correspondence relationship between the object Ot-τand the object Ot. For example, the affinity matrix AM is an information that indicates (1) a first object Ot-τof the plurality of objects Ot-τcorresponds to a first object Otof the plurality of objects Ot(namely, both are the same objects), (2) a second object Ot-τof the plurality of objects Ot-τcorresponds to a second object Otof the plurality of objects Ot-τ. . . , and (N) a N-th object Ot-τof the plurality of objects Ot-τcorresponds to a N-th object Otof the plurality of objects Ot. Note that the affinity matrix AM may be referred to as a correspondence information, because it is the information indicating a correspondence relationship between the object Ot-τand the object Ot.

Specifically, as illustrated inFIG.5, the affinity matrix AM may be regarded to be a matrix in which a vertical axis corresponds to a vector component of the feature vector CVt-τand a horizontal axis corresponds to a vector component of the feature vector CVt. Thus, a size of the vertical axis of the affinity matrix is HW (namely, the size of the feature vector CVt-τ, and a size corresponding to a size (namely, the number of the pixel) of the image IMt-τcaptured at the time t-τ. Similarly, a size of the horizontal axis of the affinity matrix is HW (namely, the size of the feature vector CVt, and a size corresponding to a size (namely, the number of the pixel) of the image IMtcaptured at the time t. In other words, the affinity matrix AM may be regarded to be a matrix in which the vertical axis corresponds to the detected result of the object Ot-τ(namely, a detected position of the object Ot-τ) included in the image IMt-τat the time t-τ and the horizontal axis corresponds to the detected result of the object Ot(namely, a detected position of the object Ot) included in the image IMtat the time t. In this case, an element of the affinity matrix AM reacts (typically, has a value that is not zero) at a position at which a vector component corresponding to a certain object Ot-τon the vertical axis intersects with a vector component corresponding to the same object Oton the horizontal axis. In other words, the element of the affinity matrix AM reacts at a position at which the detected result of the object Ot-τon the vertical axis intersects with the detected result of the object Oton the horizontal axis. Namely, the affinity matrix AM is typically a matrix in which a value of the element at the position at which the vector component corresponding to a certain object Ot-τincluded in the feature vector CVt-τintersects with the vector component corresponding to the same object Otincluded in the feature vector CVtis a value that is obtained by multiplying both vector components (namely, a value that is not zero) and a value(s) of the other element(s) is zero.

For example, in an example illustrated inFIG.5, the element of the affinity matrix AM reacts at a position at which the vector component corresponding to an object O #k (note that k is the number of the detected object O, and k=1, 2, 3 or 4 in the example illustrated inFIG.5) included in the feature vector CVt-τintersects with the vector component corresponding to the same object O #k included in the feature vector CVt. Namely, the element of the affinity matrix AM reacts at a position at which the detected result of the object O #k included in the image IMt-τintersects with the detected result of the object O #k included in the image IMt.

Conversely, when the element of the affinity matrix AM does not reacts (typically, is zero) at the position at which the vector component corresponding to a certain object Ot-τincluded in the feature vector CVt-τintersects with the vector component corresponding to the same object Otincluded in the feature vector CVt, it is presumed that the object Ot-τincluded in the image IMt-τis not included in the image IMt(for example, goes out of an imaging angle of view of the camera).

As described above, the affinity matrix AM is usable as the information indicating the correspondence relationship between the object Ot-τand the object Ot. Namely, the affinity matrix AM is usable as the information indicating the matching result of the object Ot-τincluded in the image IMt-τand the object Otincluded in the image IMt. The affinity matrix AM is usable as an information for tracking the position of the object Ot-τ, which has been included in the image IMt-τin the image IMt.

The information indicating the affinity matrix AM generated by the matrix calculation unit225is stored in the object matching DB33as the object matching information indicating the result of the object matching operation by the object matching unit22. Alternatively, the matrix calculation unit225may generate, based on the affinity matrix AM, another type of information indicating the correspondence relationship between the object Ot-τand the object Ot, and another type of information indicating the correspondence relationship between the object Ot-τand the object Ot, may be stored in the object matching DB33as the object matching information.

(2-3) Refinement Operation

Next, the refinement operation performed by the refinement unit23that is one specific example of a “correction unit” will be described. The refinement unit23reads the object position information PI stored in the object detection DB32, obtains the affinity matrix AM from the object matching unit22, and corrects the read object position information PI by using the obtained affinity matrix AM. In the below described description, with reference toFIG.6, the refinement operation for correcting the object position information PItby using the affinity matrix AM that is generated based on the object position information PIt-τand PItwill be described.FIG.6is a flowchart that illustrates a flow of the refinement operation.

As illustrated inFIG.6, the feature map conversion unit236obtains (namely, reads) the object position information PItstored in the object detection DB32(a step S231). Then, the feature map conversion unit236generates a feature map CM′t from the object position information PItobtained at the step S231(a step S232). Here, as described above, the feature map conversion unit223of the object matching unit22also generates the feature map CMtfrom the object position information PIt. However, the feature map conversion unit223generates the feature map CMtfor the purpose of generating the affinity matrix AM (namely, for the purpose of performing the object matching operation). The feature map conversion unit223generates the feature map CMtfor the purpose of generating the affinity matrix AM. On the other hand, the feature map conversion unit236generates the feature map CM′tfor the purpose of correcting the object position information PItby using the affinity matrix AM generated by the object matching operation (namely, for the purpose of performing the refinement operation). Thus, the feature map conversion units223and236learned (namely, updated) by a below described learning operation are different from each other in that the feature map conversion unit223generates the feature map CMtthat is more suitable for generating the affinity matrix AM and the feature map conversion unit236generates the feature map CM′t that is more suitable for correcting the object position information PIt. Namely, the feature maps CMtand CM′tare different from each other in that the feature map CMtis more suitable for generating the affinity matrix AM and the feature map CM′tis more suitable for correcting the object position information PIt. Thus, the refinement unit23includes the feature map conversion unit236separately from the feature map conversion unit223of the object matching unit22. Note that the feature map conversion units223and236are built as a result of the below described learning operation. Namely, the below describe learning operation allows the feature map conversion unit223to be learned so as to generate the feature map CMtthat is more suitable for generating the affinity matrix AM and allows the feature map conversion unit236to be learned so as to generate the feature map CM′tthat is more suitable for correcting the object position information Pit.

The feature map conversion unit236may generate the feature map CM′tby using a calculation model that is configured to output the feature map CM when the object position information PI is inputted thereto. One example of the calculation model is a calculation model using a Neural Network (for example, a CNN: Convolutional Neural Network). In this case, a learning of the calculation model may be performed by using learning data, as described later. Namely, a parameter of the calculation model may be optimized so as to output the proper feature map CM (especially, for correcting the object position information PIt).

Then, the feature vector conversion unit237generates a feature vector CV′tfrom the feature map CM′tgenerated at the step S232(a step S233).

In parallel with, after or before the processing from the step S231to the step S233, the matrix calculation unit231obtains the affinity matrix AM generated by the object matching unit22from the object matching unit22(a step S234). Alternatively, the matrix calculation unit231may obtain the affinity matrix AM generated by the object matching unit22from the object matching DB33(the step S234).

Then, the matrix calculation unit231generates a feature vector CV_res by using the feature vector CV′tgenerated at the step S233and the affinity matrix AM obtained at the step S234(a step S235). Specifically, the matrix calculation unit231generates, as the feature vector CV_res, an information that is obtained by a calculation processing using the feature vector CV′tand the affinity matrix AM.

For example, the matrix calculation unit231may generate, as the feature vector CV_res, an information that is obtained by a calculation processing for calculation a matrix product of the feature vector CV′tand the affinity matrix AM (namely, the matrix product of the feature vector CV′tand the affinity matrix AM).

Then, the feature vector conversion unit232generates a feature map CM res having a size that is same as that of the feature map CM from the feature vector CV_res generated at the step S235(a step S236). Namely, the feature vector conversion unit232generates the feature map CM_res having a size of H×W×C from the feature vector CV_res having any size. For example, the feature vector conversion unit232may generate the feature map CM_res by converting the feature vector CV_res into the feature map CM_res.

Then, the feature map conversion unit233generates the object position information PItres having a size that is same as that of the object position information PItfrom the feature map CM_res generated at the step S236(a step S237). Namely, the feature map conversion unit233generates the object position information PIt_res that is the map information having the size of H×W×(K+4) from the feature map CM_res generated at the step S236. The feature map conversion unit233generates, from the feature map CM_res generated at the step S236, the object position information PIt_res that includes (i) the map information that has a size of H×W×K and that indicates the center point KP of the object Otin the image IMt, (ii) the map information that has a size of H×W×2 and that indicates the size of the detection frame BB of the object Otin the image IMt, and (iii) the map information that has a size of H×W×2 and that indicates the correction amount of the detection frame BB of the object Otin the image IMt. namely, the feature map conversion unit233generates the object position information PIt_res from the feature map CM_res by converting a dimension of the feature map CM_res generated at the step S236.

For example, the feature map conversion unit233may generate the object position information PIt_res by converting the feature map CM_res into the object position information PIt_res. The feature map conversion unit233may generate the object position information PIt_res by using a calculation model that is configured to output the object position information PI when feature map CM is inputted thereto. One example of the calculation model is a calculation model using a Neural Network (for example, a CNN: Convolutional Neural Network). In this case, a learning of the calculation model may be performed by using learning data, as described later. Namely, a parameter of the calculation model may be optimized so as to output the proper object position information PIt_res.

A processing at the step S237may be regarded to be substantially equivalent to a processing for generating the object position information PIt_res by using an attention mechanism that uses the affinity matrix AM as a weight. Namely, the refinement unit23may be regarded to constitute at least a part of the attention mechanism including the matrix calculation unit231, the feature vector conversion unit232and the feature map conversion unit233. The object position information PI res may be used as the refined object position information PIt. In this case, the processing at the step S237may be regarded to be substantially equivalent to a processing for correcting (in other words, updating, adjusting or improving) the object position information PItby using the attention mechanism that uses the affinity matrix AM as the weight.

However, there is a possibility that an information included in the original object position information PIt(namely, the object position information PIton which the refinement operation is not performed) is lost from the object position information PIt_res generated at the step S235. This is because the affinity matrix AM that indicates a part to which the attention mechanism should pay attention (in the present example embodiment, the detected position of the object Ot) is used as the weight, and thus, there is a possibility that an information part of the object detection information that is different from an information related to the detected position of the object Otis lost in the object position information PItres. Thus, in the present example embodiment, the refinement unit23may further perform a processing for preventing the loss of the information included in the original object position information PIt. Specifically, the residual processing unit234may generate an object position information PIt_ref by adding the object position information PIt_res generated at the step S237to the original object position information PIt(a step S238). The object position information PIt_ref has a size that is same as that of the object position information PIt. Thus, the residual processing unit234generates the object position information PIt_ref that is the map information having a size of H×W×(K+4). In order to generate the object position information PIt_ref that is the map information having the size of H×W×(K+4), the residual processing unit234(i) adds the map information indicating the center point KP of the object Otincluded in the object position information PIt_res and the map information indicating the center point KP of the object Otincluded in the original object position information PIt, (ii) adds the map information indicating the size of the detection frame BB of the object Otincluded in the object position information PIt_res and the map information indicating the size of the detection frame BB of the object Otincluded in the original object position information PIt, and (iii) adds the map information indicating the correction amount of the detection frame BB included in the object position information PIt_res and the map information indicating the correction amount of the detection frame BB included in the original object position information PIt.

In this case, a processing at the step S238may be regarded to be substantially equivalent to a processing for generating the object position information PIt_ref by using a residual attention mechanism including the residual processing unit234. Namely, the refinement unit23may be regarded to constitute at least a part of the residual attention mechanism including the matrix calculation unit231, the feature vector conversion unit232, the feature map conversion unit233and the residual processing unit234. The object position information PIt_ref is the refined object position information PItand includes the information included in the original object position information Pit. In this case, the object position information PIt_ref may be used as the refined object position information PIt. However, the refinement unit23may not perform the processing for preventing the loss of the information included in the original object position information PIt(the processing at the step S238). In this case, the refinement unit23may not include the residual processing unit234.

Furthermore, the feature map conversion unit235may generate, from the object position information PIt_ref, an object position information PIt_ref′ having a size: H×W×(K+4) that is same as that of the object position information PIt_ref (a step S239). The object position information PIt_ref′ has the size that is same as that of the object position information PIt. Thus, the feature map conversion unit234generates the object position information PI_ref′ that is the map information having the size: H×W×(K+4). Typically, the feature map conversion unit235may generate the object position information PIt_ref′ by performing a convolution processing (for example, a processing performed at a convolution layer of a Neural Network) on the object position information PIt_ref. The feature map conversion unit235may convert the object position information PIt_ref into the object position information PIt_ref′ by inputting the object position information PIt_ref into the convolution layer of the Neural Network. In this case, the object position information PIt_ref′ may be used as the refined object position information PIt. However, the refinement unit23may not include the feature map conversion unit235.

When the object position information PItres, the object position information PIt_ref or the object position information PItref′ is used as the refined object position information PIt, the object position information PItstored in the object detection DB32may be replaced by the object position information PItres, the object position information PIt_ref or the object position information PIt_ref′. In this case, the object matching unit22is capable of performing the object matching operation for performing the matching between the object Otincluded in the image IMtcaptured at the time t and an object Ot+τincluded in an image IMt+τcaptured at a time t+τ by using the refined object position information PIt, instead of the not-refined object position information PIt. furthermore, an object position information PIt+τrelated to the object Ot+τis refined by using the result of the object matching operation for performing the matching between the object Otand the object Ot+τ. Thus, the object matching unit22is capable of performing the object matching operation for performing the matching between the object Ot+τincluded in the image IMt+τcaptured at the time t+τ and an object Ot+2τincluded in an image IMt+2τ captured at a time t+2τ by using the refined object position information PIt+τ, instead of the not-refined object position information PIt+τ. Namely, when the object position information PItstored in the object detection DB32is replaced by the object position information PIt_res, the object position information PIt-τref or the object position information PIt_ref′, the refined object position information PI is sequentially inputted into the object matching unit22. Thus, an improvement of an accuracy of the matching of the object O and a reduction of a processing cost necessary for the object tracking operation are expected, compared to a case where the object matching operation is performed by using the not-refined object position information PI.

(3) Technical Effect of Object Tracking Apparatus1

As described above, the object tracking apparatus1in the present example embodiment is capable of generating the affinity matrix AM from the object position information PIt-τand PIt, and refining the object position information PItby using the affinity matrix AM. Namely, the object tracking apparatus1is capable of performing the object matching operation without performing an pre-processing on the object position information PIt-τand PItthat are outputs of the object detection unit21. In other words, the object tracking apparatus1is capable of using, as an input to the object matching unit22that performs the object matching operation, the object position information PIt-τand PItthat are the outputs of the object detection unit21. Similarly, the object tracking apparatus1is capable of performing the refinement operation without performing an pre-processing on the affinity matrix AM that is an output of the object matching unit22. In other words, the object tracking apparatus1is capable of using, as an input to the refinement unit23that performs the refinement operation, the affinity matrix AM that is the output of the object matching unit22and the object position information PI that is the output of the object detection unit21. Thus, the object tracking operation does not necessarily perform the pre-processing and the like for performing each of the object detection operation, the object matching operation and the refinement operation, compared to an object tracking apparatus in a comparison example that performs the object detection operation, the object matching operation and the refinement operation as three separate and independent operations As a result, the object tracking apparatus1is capable of tracking the object O with relatively low processing cost.

Incidentally, it can be said that the object tracking apparatus performs the refinement operation by using an information generated by the object matching operation by considering that the object tracking operation is similar to an operation performed by a general attention mechanism. Specifically, in the object tracking operation, a processing for detecting the object O, a processing for performing a matching of the object O and a processing for refining a detected result of the object O are performed, as described above. On the other hand, in the general attention mechanism, a processing for extracting a feature of the object O, a processing for calculating a weight and a processing for refining an extracted result of the feature of the object O are performed. The object tracking apparatus1substantially uses the processing for calculating the weight in the attention mechanism as the processing for performing the matching of the object O in the object tracking operation. In other words, the object tracking apparatus1substantially uses the processing for performing the matching of the object O in the object tracking operation as the processing for calculating the weight in the attention mechanism. Therefore, it can be said that the object tracking apparatus1realizes the object detection operation, the object matching operation and the refinement operation by using the attention mechanism. Specifically, it can be said that the object tracking apparatus1performs the object tracking operation by using the attention mechanism that uses the object position information PI-τas a query, the object position information PItas a key and a value, and the affinity matrix AM as the weight. Thus, the object tracking apparatus1is capable of performing, as a series of operation, the object tracking operation including the object detection operation, the object matching operation and the refinement operation. Namely, the object tracking apparatus1is capable of performing the object tracking operation by using a single network structure (what we call an End to End network structure in a single stage type) that performs the object detection operation, the object matching operation and the refinement operation. Since the object tracking operation (especially, the refinement operation) can be performed by using the attention mechanism, the object tracking apparatus1is capable of refining the object position information PI more properly, compared to a case where the object tracking operation is performed without using the attention mechanism (namely, a network structure the performs the object detection operation, a network structure the performs the object matching operation a network structure the performs the refinement operation are used separately and independently).

Moreover, when an object the number of which is Nt-τis included in the image captured at the time t-τ and an object the number of which is Ntis included in the image captured at the time t, the object tracking apparatus in the comparison example (especially, the object tracking apparatus that performs the object matching operation without generating the affinity matrix AM) needs to perform the matching between each of the Nt-τobject and each of the Ntobject individually. Namely, the object tracking apparatus needs to repeat the operation for performing the matching between two objects by Nt-τ×Nttimes. Thus, there is a possibility that the processing cost for tracking the object is high, which is a technical problem. On the other hand, the object tracking apparatus1in the present example embodiment is capable of performing the object matching operation for performing the matching between the object Ot-τincluded in the image IMt-τcaptured at the time t-τ and the object Otincluded in the image IMtcaptured at the time t by using the feature vectors CVt-τand CVt. Thus, the object tracking apparatus1is capable of tracking the object O with relatively low cost. Specifically, even when the object O the number of which is Nt-τ(note that Nt-τis an integer equal to or larger than 1) is included in the image IMt-τand the object O the number of which is Nt(note that Ntis an integer equal to or larger than 1) are included in the image IMt, the object tracking apparatus1does not necessarily perform the matching between each of the Nt-τobject Ot-τand each of the Ntobject Otindividually. Namely, the object tracking apparatus1does not necessarily repeat the operation for performing the matching between two objects Ot-τand Otby Nt-τ×Nttimes. Namely, the object tracking apparatus1is capable of completing the matching between each of the Nt-τobject Ot-τand each of the Ntobject Otby performing the processing for generating the affinity matrix AM by using the feature vectors CVt-τand CVtat once. Thus, it is possible to track the object O with relatively low processing cost compared to the object tracking apparatus in the comparison example. Especially, although the processing cost increases exponentially as the number of the object O included in the image IM increases in the object tracking apparatus in the comparison example, the processing cost is less likely to depend on the number of the object O included in the image IM in the object tracking apparatus1in the present example embodiment. Thus, an effect of reduction of the processing cost by the object tracking apparatus1is more remarkable as the number of the object O included in the image IM increases.

(4) Modified Example

(4-1) First Modified Example

Firstly, with reference toFIG.7, an object tracking apparatus1ain a first modified example will be described.FIG.7is a block diagram that illustrates a configuration of the object tracking apparatus1ain the first modified example.

As illustrated inFIG.7, the object tracking apparatus1ain the first modified example is different in that it includes a learning unit24athat is configured to perform the learning operation for the object detection unit21, the object matching unit22and the refinement unit23as a logical functional block implemented in the arithmetic apparatus2. Furthermore, the object tracking apparatus1ais different in that the storage apparatus3stores a learning DB34afor storing learning data341afor the learning operation. A residual feature of the object tracking apparatus1amay be same as residual feature of the object tracking apparatus1. Note that the learning operation is typically performed before the object tracking apparatus1aactually performs the object tracking operation. However, the learning operation may be performed at a desired timing after the object tracking apparatus1astarts the object tracking operation.

The learning data341ainclude the image IM obtained (in other words, captured) at a certain time, for example, as illustrated inFIG.8. Furthermore, the learning data341ainclude an object position information PI label that indicates a ground truth label of the of the position of the object O included in the image IM. In an example illustrated inFIG.8, the learning DB34astores a plurality of learning data341aincluding: the learning data341athat includes an image IMsobtained at a time s and an object position information PIs_label related to the position of an object Osincluded in the image IMs; and the learning data341athat includes an image IMs+τobtained at a time s+τ and an object position information PIs+τ_label related to the position of an object Os+τincluded in the image IMs+τ. Incidentally, when the plurality of object O are included in each image IM, the learning data341amay include a plurality of object position information PI_label that indicate the ground truth labels of the positions of the plurality of objects O, respectively.

In order to perform the learning operation, the learning unit24ainputs the images IMsand IMs+τinclude in the learning data341ainto the object detection unit21. As a result, the object detection unit21outputs an object position information PIsrelated to the position of the object Osand an object position information PIs+τrelated to the position of the object Os+τ. Furthermore, the object matching unit22generates the affinity matrix AM by using the object position information PIsand the object position information PIs+τ. Furthermore, the refinement unit23refines t the object position information PIs+τby using the affinity matrix AM.

The learning unit24aupdates, based on at least a loss function L1related to the refined object position information PIs+τ, the learnable calculation model that is used by the object detection unit21to perform the object detection operation, is used by the object matching unit22to perform the object matching operation and/or is used by the refinement unit23to perform the refinement operation. Namely, the learning unit24aupdates, based on the loss function L1, the calculation model that defines a detail of an operation of at least one of the object detection unit21, the object matching unit22and the refinement unit23. As one example, when the loss function L1is a loss function that decreases as a difference between the refined object position information PIs+τand the object position information PIs+τ_label that is the ground truth label decreases, the learning unit24amay update the calculation model so that the loss function L1decreases (typically, is minimized).

In the present example embodiment, the object detection unit21, the object matching unit22and the refinement unit23can be realized by the single network structure (namely, a single calculation model) as described above. Thus, when the object detection unit21, the object matching unit22and the refinement unit23can be realized by the single network structure (namely, the single calculation model), the learning unit24amay update the single calculation unit. Note that one example of the calculation model is a calculation model using a Neural Network (for example, a CNN: Convolutional Neural Network). In this case, an operation for updating the calculation model may include an operation for updating, determining or adjusting a parameter of the calculation model. The parameter of the calculation model may include at least one of a weight between nodes of the Neural Network, a bias added at each node and a connection path between the nodes.

The learning unit24arepeats the learning operation by inputting the plurality of images IM, which are respectively included in the plurality of learning data341astored in the learning DB34a, into the object detection unit21in sequence in an order of the time corresponding to the plurality of images IM. Namely, the learning unit24ainputs the images IMsand IMs+τinto the object detection unit21, and updates the calculation model based on the loss function L1related to the refined object position information PIs+τ. Then, the learning unit24ainputs the images IMs+τand IMs+2τinto the object detection unit21, and updates the calculation model based on the loss function L1related to the refined object position information PIs+2τ. Then, the learning unit24arepeats the same operation. As a result, the calculation model that defines the detail of the operation of at least one of the object detection unit21, the object matching unit22and the refinement unit23is properly updated (namely, learned).

The learning unit24amay vary a temporal interval between two times each of which corresponds to respective one of two images IM inputted to the object detection unit21. For example, in the above described description, the learning unit24ainputs, to the object detection unit21, two images IM each of which is obtained at respective one of two times that are different from each other by the temporal interval of “τ” (for example, the images IMsand IMs+τ). However, the learning unit24amay input, to the object detection unit21, two images IM each of which is obtained at respective one of two times that are different from each other by the temporal interval of “m×τ” (for example, the images IMsand IMs+mτ) (note that m is a coefficient that is variable by the learning unit24a, and is an integer such as 1, 2, 3, . . . , for example). In this case, the learning unit24amay input the images IMsand IMs+mτinto the object detection unit21, and updates the calculation model based on the loss function L1related to the refined object position information PIs+mτ. Then, the learning unit24amay input the images IMs+mτand IMs+2mτinto the object detection unit21, and updates the calculation model based on the loss function L1related to the refined object position information PIs+2mτ. Then, the learning unit24arepeats the same operation. In this case, a moving distance of the object O between two images IM inputted to the object detection unit21varies depending on the coefficient m. The coefficient m may be determined by a random number, for example, for every learning operation, namely, every time the learning unit24ainputs two images IM (for example, the images IMsand IMs+τ) into the object detection unit21. As a result, the calculation model that defines the detail of the operation of at least one of the object detection unit21, the object matching unit22and the refinement unit23is updated so that the objects moving at various velocities can be traced.

The learning unit24amay updates the calculation model based on not only the loss function L1related to the refined object position information PIs+τbut also a loss function L2related to the object position information PIsoutputted by the object detection unit21and a loss function L3related to the object position information PIs+τoutputted by the object detection unit21. The loss function L2may be a loss function that decreases as a difference between the object position information PIsoutputted by the object detection unit21and the object position information PIs_label that is the ground truth label decreases, for example. The loss function L3may be a loss function that decreases as a difference between the object position information PIs+τoutputted by the object detection unit21and the object position information PIs+τ_label that is the ground truth label decreases, for example. In this case, the learning unit24amay update the calculation model so that a total sum of the loss functions L1to L3decreases (typically, is minimized).

When the loss functions L1to L3are used, the learning unit24amay perform a weighting processing on the loss functions L1to L3, and update the calculation model based on the loss functions L1to L3on which the weighting processing is already performed. Namely, the learning unit24may perform the weighting processing on each of the loss functions L1to L3by using respective one of weighting coefficients α1, α2and α3, and update the calculation model based on a loss function that is specified by an equation of α1×L1+α2×L2+α3×L3.

As one example, each of the loss functions L1and L3are loss functions related to the object position information PIs+τand the loss function L2is a loss function related to the object position information PIs. From a viewpoint of appropriately updating the calculation model, it is preferable that a contribution of the object position information PIs(namely, a contribution of the loss functions L1and L3) to the update of the calculation model be not significantly different from a contribution of the object position information PIs+τ(namely, the contribution of the loss function L2) to the update of the calculation model. Typically, it is preferable that the contribution of the object position information PIsto the update of the calculation model be equal to the contribution of the object position information PIs+τto the update of the calculation model. Thus, the learning unit24amay perform the weighting processing so that a weight for a total sum of the loss functions L1and L3is equal to a weight for the loss function L2. As one example, the learning unit24amay perform the weighting processing on the loss functions L1to L3by using an equation of 0.5×(L1+L3)+0.5×L2, and update the calculation model based on a loss function that is specified by an equation of 0.5×(L1+L3)+0.5×L2. This weighting processing is particularly beneficial when the affinity matrix AM is normalized by using a softmax function in the third modified example described below. The following are the reasons for this. Firstly, when the affinity matrix AM is normalized by using the softmax function in an early stage of the learning operation, most of the elements of the affinity matrix AM are zero. As a result, the refined object position information PIs+τis identical to the object position information PIs+τdetected by the object detection unit21. Assuming that the weights of the loss functions L1to L3are the same (for example, all are 1) under such situation, a learning effect of the calculation model is reduced. This is because values of the loss functions L1and L3are the same, and thus, the contribution of the loss function L2to the update of the calculation model is apparently smaller than the contribution of the loss functions L1and L3to the update of the calculation model. Thus, the learning unit24amay perform the weighting processing so that the weight for the total sum of the loss functions L1and L3is equal to the weight for the loss function L2in the early stage of the learning operation (namely, in an period when most of the elements of the affinity matrix AM are zero by a normalization processing). As a result, the learning unit24ais less likely to lose its learning effect even in the early stage of the learning operation (namely, in an period when most of the elements of the affinity matrix AM are zero by a normalization processing).

Incidentally, the learning unit24amay perform the weighting processing on the loss functions L1to L3by using an equation of 0.25×L1+0.25×L3+0.5×L2, and update the calculation model based on a loss function that is specified by an equation of 0.25×L1+0.25×L3+0.5×L2, in order to allow the contribution of the loss function L1to the update of the calculation model to be equal to the contribution of the loss function L3to the update of the calculation model.

(4-2) Second Modified Example

In the above described description, the object tracking apparatus1performs the object tracking operation including the object detection operation. However, in the second modified example, the object tracking operation may not include the object detection operation. Namely, the object tracking apparatus1may not perform the object detection operation. In this case, as illustrated inFIG.9that illustrates a configuration of an object tracking apparatus1bthat is a second modified example of the object tracking apparatus1, the object tracking apparatus1bmay not include the object detection unit21. Furthermore, the storage apparatus3may not store the image DB31for storing the image IM used to perform the object detection operation and the object detection DB32for storing the object detection information indicating the result of the object detection operation. The object tracking apparatus1b(especially, the object matching unit22) may perform the object matching operation by using the object detection information that indicates the result of the object detection operation performed by an apparatus different from the object tracking apparatus1b.

In the above described description, the object tracking apparatus1performs the object tracking operation including the refinement operation. However, in the second modified example, the object tracking operation may not include the refinement operation. Namely, the object tracking apparatus1may not perform the refinement operation. In this case, as illustrated inFIG.9that illustrates the configuration of the object tracking apparatus1bthat is the second modified example of the object tracking apparatus1, the object tracking apparatus1bmay not include the refinement unit23. The refinement operation may be performed by an apparatus different from the object tracking apparatus1b. In this case, the object tracking apparatus1b(especially, the object matching unit22) may output the object matching information indicating the result of the object matching operation to an apparatus that performs the refinement operation and that is different from the object tracking apparatus1b.

(4-3) Third Modified Example

The matrix calculation unit225may normalize the affinity matrix AM obtained by the calculation processing using the feature vector CVt-τand the feature vector CVt. For example, the matrix calculation unit225may normalize the affinity matrix AM by normalizing the matrix product of the feature vector CVt-τand the feature vector CVt.

The matrix calculation unit225may perform any normalization processing on the affinity matrix AM. For example, the matrix calculation unit225may perform a normalization processing using a sigmoid function on the affinity matrix AM. In this case, each element of the affinity matrix AM is normalized using the sigmoid function.

Alternatively, for example, the matrix calculation unit225may perform normalization processing using a softmax function on the affinity matrix AM. Specifically, as illustrated inFIG.10, the matrix calculation unit225may perform the normalization processing using the softmax function on each of the column vector component including a plurality of elements in each row of the affinity matrix AM and the column vector component including a plurality of elements in each column of the affinity matrix AM. In this case, the matrix calculation unit225may perform the normalization processing using the softmax function on the column vector component so that a total sum of the plurality of elements included in the column vector component is 1, and perform the normalization processing using the softmax function on the row vector component so that a total sum of the plurality of elements included in the row vector component is 1. Then, a matrix that includes the elements obtained by multiplying the vector component with the row vector component on which the normalization processing is already performed is the normalized affinity matrix AM. As one example, the matrix calculation unit225may perform the normalization processing using the softmax function on each of the column vector component including the plurality of elements in one row of the affinity matrix AM corresponding to the object Ot-τand the column vector component including the plurality of elements in one column of the affinity matrix AM corresponding to the same object Ot. In this case, the matrix calculation unit225the matrix calculation unit225may perform the normalization processing using the softmax function on the column vector component so that a total sum of the plurality of elements included in the column vector component corresponding to the object Ot-τis 1, and perform the normalization processing using the softmax function on the row vector component so that a total sum of the plurality of elements included in the row vector component corresponding to the object Otis 1. Then, a matrix that includes elements obtained by multiplying the vector component with the row vector component on which the normalization processing is already performed is the normalized affinity matrix AM.

(4-4) Other Modified Example

In the above described description, the object position information PI generated by the object detection unit21performing the object detection operation includes the information indicating the center point KP of the object O, the information indicating the size of the detection frame BB of the object O and the information indicating the correction amount of the detection frame BB. However, the object position information PI may not include at least one of the information indicating the size of the detection frame BB of the object O and the information indicating the correction amount of the detection frame BB. The object position information PI may include an information indicating a position of a part of the object O that is different from the center.

In the above described description, the object matching unit22generates the feature vector CV by using the object position information PI indicating the result of the object detection operation. Here, when the calculation model such as the Neural Network is used at the object detection operation as described above, the object matching unit22may generate the feature vector CV by using an intermediate output of the calculation model in addition to the object position information that is a final output of the calculation model. For example, the object matching unit22may generate the feature map CM by using an output of a hidden layer of the Neural Network that is used as the calculation model in addition to an output of an output layer of the Neural Network that is used as the calculation model, and then generate the feature vector CV from the generated feature map CM.

In the above described description, the matrix calculation unit225generates, as the affinity matrix AM, the information that is obtained by the calculation processing for calculating the matrix product of the feature vector CVt-τand the feature vector CVt. However, the matrix calculation unit225may generate, as the affinity matrix AM, an information that is obtained by any calculation processing using the feature vector CVt-τand the feature vector CVt. For example, the matrix calculation unit225may generate, as the affinity matrix AM, an information that is obtained by the calculation processing for calculating the matrix sum of the feature vector CVt, and the feature vector CVt. For example, the matrix calculation unit225may generate, as the affinity matrix AM, by using a calculation model that is configured to output the affinity matrix AM when the feature vector CVt-τand the feature vector CVtare inputted thereto. One example of the calculation model is a calculation model using a Neural Network (for example, a CNN: Convolutional Neural Network). It follows that the matrix calculation unit225may perform any calculation processing to generate the affinity matrix AM, as long as it performs the calculation processing that is capable of generating the affinity matrix AM indicating the correspondence relationship between the object Ot-τand the object Ot.

In the above described description, the object matching unit22generates the affinity matrix AM having the size of HW×HW from the object position information PI that is the map information having the size of H×W×(K+4). However, the object matching unit22generates the affinity matrix AM having a size that is smaller than the size of HW×HW from the object position information PI having the size of H×W×(K+4). Namely, the object matching unit22may generate the downscaled affinity matrix AM. For example, the feature map conversion units221and223of the object matching unit22may generate the feature maps CMt-τand CMteach of which has a size that is smaller than the size of H×W×C from the object position information PIt-τand PIteach of which has the size of H×W×(K+4). In this case, the matrix calculation unit225of the object matching unit22is capable of generates the affinity matrix AM having the size that is smaller than the size of HW×HW from the feature map CM having the size that is smaller than the size of H×W×C. Incidentally, in order to generate the feature maps CMt-τand CMteach of which has the size that is smaller than the size of H×W×C, the feature map conversion units221and223of the object matching unit22may adjust a stride amount (namely, a moving distance) of a kernel (namely, a convolution filter) used in the convolution layer that performs the convolution processing for generating the feature maps CMt-τand CMt. As the stride amount of the kernel increases, the size of each of the generated feature maps CMt-τand CMtdecreases.

(5) Supplementary Note

With respect to the example embodiments described above, the following Supplementary Notes will be further disclosed.

[Supplementary Note 1]

An object tracking apparatus including:a first generation unit that is configured to generate, based on a first position information related to a position of an object in a first image captured at a first time and a second position information related to a position of an object in a second image captured at a second time that is different from the first time, a first feature vector representing a feature of the first position information and a second feature vector representing a feature of the second position information; anda second generation unit that is configured to generate, as a correspondence information indicating a correspondence relationship between the object in the first image and the object in the second image, an information that is obtained by a calculation processing using the first and the second feature vectors.
[Supplementary Note 2]

The object tracking apparatus according to Supplementary Note 1, whereinthe calculation processing includes a processing for calculating a matrix product of the first feature vector and the second feature vector.
[Supplementary Note 3]

The object tracking apparatus according to Supplementary Note 1 or 2, whereinthe correspondence information indicates the correspondence relationship between the object in the first image and the object in the second image by using a matrix.
[Supplementary Note 4]

The object tracking apparatus according to any one of Supplementary Notes 1 to 3, whereinthe second generation unit is configured to normalize the matrix by normalizing each of a vector component in one row of the matrix and a vector component in one column of the matrix by a softmax function.
[Supplementary Note 5]

The object tracking apparatus according to any one of Supplementary Notes 1 to 4, whereinthe correspondence information indicates the correspondence relationship between the object in the first image and the object in the second image by using a matrix in which an element react at a position at which a row vector component corresponding to one object in either one of the first and second images intersects with a column vector component corresponding to the one object in the other one of the first and second images,the second generation unit is configured to normalize the matrix by normalizing each of the row vector component and each column vector component by a softmax function.
[Supplementary Note 6]

The object tracking apparatus according to any one of Supplementary Notes 1 to 4 further including a correction unit that is configured to correct the second position information by using the correspondence information.

[Supplementary Note 7]

The object tracking apparatus according to Supplementary Note 6, whereinthe correction unit is configured to correct the second position information by using an attention mechanism that uses the correspondence information as a weight.
[Supplementary Note 8]

The object tracking apparatus according to Supplementary Note 7, whereinthe correspondence information indicates the correspondence relationship between the object in the first image and the object in the second image by using a matrix.the second position information includes a position map indicating an information related to a position of the object in the second image,the attention mechanism is configured to correct the position map that is the second position information by performing a processing for calculating a matrix product of the position map and the correspondence map.
[Supplementary Note 9]

The object tracking apparatus according to Supplementary Note 8, whereinthe attention mechanism is configured to correct the position map that is the second position information by performing a processing for adding a correction map, which is obtained by calculating the matrix product of the position map and the correspondence information, to the position map.
[Supplementary Note 10]

The object tracking apparatus according to any one of Supplementary Notes 1 to 9, whereinthe first generation unit is configured to obtain, from a calculation model that outputs each of the first and second position information when the first and second images are inputted thereto, the first and second position information and an intermediate output information corresponding to an intermediate output of the calculation model,the first generation unit is configured to generate the first and second feature vectors based on the first and second position information and the intermediate output information.
[Supplementary Note 11]

The object tracking apparatus according to any one of Supplementary Notes 1 to 10 further including:a third generation unit that is configured to generate the first and second position information from the first and second images;a correction unit that is configured to correct the second position information by using the correspondence information; anda learning unit that is configured to input, into the third generation unit, learning data including the first and second images for a learning, and to update a learning model that defines a detail of an operation of at least one of the first to third generation units and the correction unit based on a first loss function related to the second position information corrected by the correction unit when the learning data is inputted to the third generation unit.
[Supplementary Note 12]

The object tracking apparatus according to Supplementary Note 11, whereinthe learning unit is configured to update the learning model based on the first loss function, a second loss function related to the first position information generated by the third generation unit when the learning data is inputted to the third generation unit and a third loss function related to the second position information generated by the third generation unit when the learning data is inputted to the third generation unit,the learning unit is configured to perform a weighting processing on the first to third loss functions so that a weight of whole of the first and third loss functions is equal to a weight of the second loss function, and to generate the learning model based on the first to third loss functions on which the weighting processing is already performed.
[Supplementary Note 13]

The object tracking apparatus according to Supplementary Note 11 or 12, whereinthe learning unit is configured to input, into the first generation unit, a plurality of learning data sets having different intervals between the first time and the second time.
[Supplementary Note 14]

An object tracking method including:generating, based on a first position information related to a position of an object in a first image captured at a first time and a second position information related to a position of an object in a second image captured at a second time that is different from the first time, a first feature vector representing a feature of the first position information and a second feature vector representing a feature of the second position information; andgenerating, as a correspondence information indicating a correspondence relationship between the object in the first image and the object in the second image, an information that is obtained by a calculation processing using the first and the second feature vectors.
[Supplementary Note 15]

Anon-transitory recording medium on which a computer program that allows a computer to execute an object tracking method is recorded,the object tracking method including:generating, based on a first position information related to a position of an object in a first image captured at a first time and a second position information related to a position of an object in a second image captured at a second time that is different from the first time, a first feature vector representing a feature of the first position information and a second feature vector representing a feature of the second position information; andgenerating, as a correspondence information indicating a correspondence relationship between the object in the first image and the object in the second image, an information that is obtained by a calculation processing using the first and the second feature vectors.
[Supplementary Note 16]

A computer program that allows a computer to execute an object tracking method is recorded,the object tracking method including:generating, based on a first position information related to a position of an object in a first image captured at a first time and a second position information related to a position of an object in a second image captured at a second time that is different from the first time, a first feature vector representing a feature of the first position information and a second feature vector representing a feature of the second position information; andgenerating, as a correspondence information indicating a correspondence relationship between the object in the first image and the object in the second image, an information that is obtained by a calculation processing using the first and the second feature vectors.

This disclosure is allowed to be changed, if desired, without departing from the essence or spirit of the invention which can be read from the claims and the entire specification, and an object tracking apparatus, an object tracking method and a recording medium, which involve such changes, are also intended to be within the technical scope of this disclosure.

DESCRIPTION OF REFERENCE CODES

1object tracking apparatus2arithmetic apparatus21object detection unit22object matching unit221,223feature map conversion unit222,224feature vector conversion unit225matrix calculation unit23refinement unit231matrix calculation unit232feature vector conversion unit233feature map conversion unit234residual processing unit235feature map conversion unit3storage unit31image DB32object detection DB33object matching DB