Patent Description:
Events such as horse races, track and field meetings, bicycle races etc. usually have an audience that want to follow the event. As technology develops and becomes more prevalent the number of applications for such technology also increases. One such area is tracking of moving objects at different events. The systems that are used today for tracking require more or less human intervention in order to function properly. Such human intervention may for example be to attach sensors to objects, identifying activities etc..

For example, <CIT> discloses methods and systems for locating an actor in an environment. For example, an information system may receive point cloud data representing a cluster of potential actors at a location in the environment. In addition, the information system may also receive location information from a device indicating that the current location of the device is proximate to the location of the cluster of potential actors.

The system in <CIT> may use different sensors as inputs, such as depth sensors, laser-based sensors and/or camera systems. When using different data sources as input, the data from the various sources may be time-stamped such that a controller may use data from the various sources in order to determine where the actor is.

The problem with <CIT> is that it is limited when it comes to handling objects with higher speed than an actor, such as race horses.

<CIT> discloses tracking of objects in order to determine the position of the object and/or the progress and performance of the object over time. The object may for example be race horses. In one embodiment a jockey wears a small infra-red beacon that is recognized by a video camera. This information is used to determine the movement and position of the race horse. The solution in <CIT> is complicated and requires human intervention when applying the infra-red beacon onto the helmet of the jockey.

Thus, there is a need of a method in which moving objects can be followed without human intervention and which method also can follow objects having a high speed. There is also a need for a controller that is configured to perform such a method.

An object of the present invention is to accomplish a method for tracking moving objects, which reduces human intervention, and which is capable of tracking objects with high speed, such as race horses.

According to one aspect of the present invention this is accomplished by a method for tracking moving objects in an environment of an event. Each moving object comprises an associated identity marking and different key portions, wherein the key portion being used to determine which moving object is first to finish, such as the nose of a horse or the chest of an athlete. The method is performed by a controller and comprises receiving images from a camera having a field-of-view in a section of the environment, identifying moving objects in the received images and separate these moving objects from a background of the environment, determining position coordinates of the associated identity markings and the different key portions of each moving object, receiving a first set of data points representing position coordinates in a field of view of a first Light Detection And Ranging, LIDAR, device, receiving a second set of data points representing position coordinates in a field of view of a second LIDAR device, wherein the field of view of the second LIDAR device is partly overlapping the field of view of the first LIDAR device, separating data points representing the moving objects from data points representing a background for each of the first and the second set of data points, modelling the position of each moving object as a two-dimensional probabilistic distribution, calculating a weighted mean value for the position coordinates of the moving objects based on the first and second set of data points giving more weight to the set of data points that represents moving objects closest to one of the first and second LIDAR device, wherein the weighted mean is calculated as inversely proportional to the variances of the moving objects, and reconstructing the position in the environment and the associated identity marking for the moving objects in a particular point of time.

In another exemplary embodiment a synchronizing signal is generated by a synchronization module and fed to the camera and to the first and second LIDAR. The synchronizing between the received images from the camera and the received first and second set of data points from the first and second LIDAR may be made with an accuracy of <NUM> to <NUM>.

According to another aspect of the present invention this is accomplished by a controller for tracking objects in an environment of an event. Each moving object comprising an associated identity marking and different key portions, wherein the key portion being used to determine which moving object is first to finish, such as the nose of a horse or the chest of an athlete, the controller comprises a processor and a non-transitory computer-readable medium, configured to store instructions, which when executed by the processor, cause the controller to receive images from a camera having a field-of-view in a section of the environment, identify moving objects in the received images and separate these moving objects from a background of the environment, determine position coordinates of the associated identity markings and the different key portions of each moving object, receive a first set of data points representing position coordinates in a field of view of a first Light Detection And Ranging, LIDAR, device, receive a second set of data points representing position coordinates in a field of view of a second LIDAR device, wherein the field of view of the second LIDAR device is partly overlapping the field of view of the first LIDAR device, separate data points representing the moving objects from data points representing a background for each of the first and the second set of data points, model the position of each moving object as a two-dimensional probabilistic distribution, calculate a weighted mean value for the position coordinates of the moving objects based on the first and second set of data points giving more weight to the set of data points that represents moving objects closest to one of the first and second LIDAR device, wherein the weighted mean is calculated as inversely proportional to the variances of the moving objects, and reconstruct the position in the environment and the associated identity marking for the moving objects in a particular point of time.

According to yet another aspect of the present invention this is accomplished by computer program comprising computer program code, wherein the computer program code is adapted, if executed on a processor, to implement the above described method. The present invention also relates to a computer program product comprising a computer readable storage medium, having the computer program stored thereon.

In the following, a detailed description of the present invention will be made using a race track for horses as an example. However, it should be understood that the present invention is applicable to a lot of different application where tracking of moving objects is desirable, such as track and field meetings, bicycle races, etc. <FIG> shows a schematic view of a race track <NUM>. In this case the race track has the shape of an oval, which may correspond to a race track for horses. The moving objects <NUM> are shown as small circles on the race track <NUM>. In order not to smudge <FIG> only one of the small circles has been donated with the reference numeral <NUM>. Along the race track there are provided a number of cameras <NUM> and a number of Light Detection And Ranging (LIDAR) devices <NUM>, <NUM>. Even though only a few cameras <NUM> and LIDAR devices <NUM>, <NUM> are shown, it should be understood that there are cameras <NUM> and LIDAR devices <NUM>, <NUM> provided all around the track <NUM>, such that the moving objects <NUM> can be tracked along the entire track.

As will be more closely described below the cameras <NUM> primarily object is to identify moving objects <NUM> and separate these moving objects <NUM> from the background of the environment, calculate coordinates of key points of the moving objects <NUM> and convert the resulting coordinates to a track coordinate system. There are today many different types of cameras <NUM> that may be used together with the present invention. It is to believe to within the skills of a person skilled in the art to select a suitable camera. Since both cameras <NUM> and LIDAR devices <NUM>, <NUM> are used there is no requirement that the cameras <NUM> cover the entire track, but they should preferably cover about <NUM>-<NUM>% of the track. Each camera <NUM> will have a fixed position and orientation and cover about <NUM> meters of track length, i.e. there will be around <NUM> cameras for covering a track of <NUM>. During use the cameras <NUM> are operable in manual focus/manual shutter speed mode, which enables catching fast moving objects <NUM>.

The primarily object of the LIDAR devices <NUM>, <NUM> is to separate moving objects points background points of the environment and convert the resulting coordinates to a track coordinate system. In one exemplary embodiment there are <NUM> LIDAR devices <NUM>, <NUM> used to cover a track of <NUM>.

Now turning to <FIG> a race horse, i.e. the moving object <NUM>, with an identity tag <NUM> and a jockey is shown. The identity tag <NUM> is preferably the starting number of the horse <NUM>. Using an already present identity tag <NUM>, will increase the efficiency of the tracking system compared to prior art, which may rely on applying infra-red beacons or the like as described in the background section. <FIG> also shows a key portion <NUM> of the race horse <NUM>, in this case the key portion <NUM> is the nose of the horse <NUM>. The key portions <NUM> may vary depending on the moving objects and may be used to determine which horse <NUM> is first to finish. If the present invention is applied in track and field the key portion may be the chest of an athlete.

<FIG> shows an overall schematic overview of a system for tracking the moving objects <NUM>. The system comprises, as mentioned above, cameras <NUM> and LIDAR devices <NUM>, <NUM>. Furthermore, the system comprises a synchronization module <NUM> and a controller <NUM>, which may be a local server on the race track on-track. The synchronization module <NUM> may use Network Time Protocol (NTP) for synchronizing all devices, i.e. the cameras <NUM>, the LIDAR devices <NUM>, <NUM> and the controller <NUM>.

The controller <NUM> is schematically shown in <FIG> and is configured to and operable for performing the method to be described in conjunction with <FIG>. The controller <NUM> comprises a processor <NUM> and a memory <NUM>. In context of the present application the term processor <NUM> should be interpreted broadly as processing circuitry, which may comprise one or more programmable processor, application-specific integrated circuits, field programmable gate arrays or combinations of these (not shown) adapted to execute instructions. The memory <NUM> contains instructions executable by said processing circuitry, whereby the controller <NUM> is operative for receiving images from the camera <NUM>, identifying moving objects <NUM> in the received images and separate these moving objects <NUM> from a background of the environment, determining position coordinates of associated identity markings <NUM> and the different key portions <NUM> of each moving object <NUM>, receiving a first set of data points representing position coordinates in a field of view of a first Light Detection And Ranging, LIDAR, device <NUM>, receiving a second set of data points representing position coordinates in a field of view of a second LIDAR device <NUM>, the field of view of the second LIDAR device <NUM> is partly overlapping the field of view of the first LIDAR device <NUM>, separating data points representing the moving objects <NUM> from data points representing a background for each of the first and the second set of data points, calculating a weighted mean value for the position coordinates of the moving objects <NUM> based on the first and second set of data points giving more weight to the set of data points that represents moving objects closest to one of the first and second LIDAR device <NUM>, <NUM>, and reconstructing the position in the environment <NUM> and the associated identity marking <NUM> for the moving objects in a particular point of time.

According to other embodiments, the controller <NUM> may further comprise an interface <NUM>, which may be considered to comprise conventional means for communicating with the cameras <NUM>, the LIDAR devices <NUM>, <NUM> and the synchronization module <NUM>. The instructions executable by the processor <NUM> may be arranged as a computer program <NUM> stored e.g. in the memory <NUM>. The processor <NUM> and the memory <NUM> may be arranged in a sub-arrangement <NUM>. The sub-arrangement <NUM> may be a micro-processor and adequate software and storage therefore, a Programmable Logic Device, PLD, or other electronic component(s)/processing circuit(s) configured to perform the methods mentioned above.

The computer program <NUM> may comprise computer readable code means, which when run in a controller <NUM> causes the controller <NUM> to perform the steps described in any of the described embodiments of the tracking method. The computer program <NUM> may be carried by a computer program product connectable to the processor <NUM>. The computer program product may be the memory <NUM>. The memory <NUM> may be realized as for example a RAM (Random-access memory), ROM (Read-Only Memory) or an EEPROM (Electrical Erasable Programmable ROM). Further, the computer program may be carried by a separate computer-readable medium <NUM>, such as a CD, DVD or flash memory, from which the program could be downloaded into the memory <NUM>. Alternatively, the computer program may be stored on a server or any other entity connected to the tracking system and has access via the interface <NUM>. The computer program may then be downloaded from the server into the memory <NUM>.

Turning now to <FIG> the tracking method of the present invention will be described in more detail. As mentioned above the tracking method is applicable for tracking moving objects <NUM> in an environment <NUM> of an event such as horse race, track and field meeting or the like. The present method makes use of any type of identity marking that is already present on the object to be tracked, such as the starting number of a horse, athlete or race car. Depending on the type of moving object to be tracked also different types key portions on the moving objects are determined.

In step S100 the controller <NUM> receives images from the camera <NUM> having a field-of-view in a section of the environment <NUM>, i.e. the race track. The controller <NUM> identifies, in step S102, the moving objects <NUM> in the received images and separates these moving objects <NUM> from the background of the environment. The controller <NUM> further determines, in step S104, the position coordinates of the associated identity markings <NUM> and the different key portions <NUM> of each moving object <NUM>, such as the start number of a race horse as identity marking and the and nose of the race horse as a key portion.

The controller <NUM> also receives, in step S106, a first set of data points representing position coordinates in a field of view of the first LIDAR device <NUM> and, in step S108, a second set of data points representing position coordinates in a field of view of the second LIDAR device <NUM>, the field of view of the second LIDAR device <NUM> is partly overlapping the field of view of the first LIDAR device <NUM>. The controller <NUM> separates, in step S110, data points representing the moving objects <NUM> from data points representing a background for each of the first and the second overlapping set of data points.

These overlapping fields may be seen as dotted lines in <FIG>. When the moving object is moving from the field of view of one LIDAR device to the field of view of another LIDAR device <NUM>, there will be a need for a seamless handover between the two LIDAR devices <NUM>, <NUM>. Thus, in context of the present description when making use of a first LIDAR device <NUM> and a second LIDAR device <NUM> it is only to denote two LIDAR devices between which hand over occurs. Thus, the second LIDAR device <NUM>, may in next instance, as the moving objects <NUM> move on, be the first LIDAR device <NUM>.

In order to ensure seamless performance of the tracking system measurements of the moving objects are taken from both LIDAR devices <NUM>, <NUM> and the controller <NUM> calculates, in step S112, a weighted mean value for the position coordinates of the moving objects <NUM> based on the first and second set of data points giving more weight to the set of data points that represents moving objects closest to one of the first and second LIDAR device <NUM>, <NUM>. The weight calculation changes over time, the closer a moving object is to one of the LIDAR devices, the more data is obtained from the closer LIDAR device for contributing to the resulting weighted mean value and the other way around. The gradual change of weights contributes to a smooth result and overcomes any possible synchronization deviations. According to the invention, the weighted mean is calculated as inversely proportional to the variances of the moving objects <NUM>.

It should be understood that one could use sets of data points from more than two LIDAR devices, depending if there are overlapping field of views or not. As for the LIDAR devices and cameras they are all active during a race and are continually transmitting data. If no moving objects are within the field of view for a specific camera or LIDAR device the values therefrom are discarded and are not included in any calculations.

The controller <NUM> then, in step S114, reconstructs the position in the environment <NUM> and the associated identity marking <NUM> for the moving objects in a particular point of time. According to the invention, the controller <NUM> performs a modeling step, S111, for modelling the position of each moving object <NUM> as a two-dimensional probabilistic distribution.

In order for the tracking system to be able to track fast moving objects with a high accuracy and no lag there will be a need to have <NUM> measurements per second to avoid a negative viewing experience. During a horse race, the horses may speed in gallop as high as <NUM>/h, i.e. about <NUM>/s, which translates to path of <NUM> passed by a horse in <NUM> (<NUM> is frame duration at <NUM> frames per second). If LIDAR device clocks are not synchronized, there will be <NUM> uncertainty as to when a measurement was taken. Thus, without an exact timestamp within a frame the positioning error can be <NUM>. Also, when horse is crossing fields of view of two different LIDARs with different errors, that horse would "blow up", i.e. one key portion of the horse, such as the nose, may be shifted by <NUM> towards a finish line (as observed by LIDAR device <NUM>), while the other part of the key portion may be shifted by <NUM> in the opposite direction (as observed by LIDAR <NUM>). Thus, in order to achieve seamless work of the tracking system, i.e. inducing an error that is less than <NUM>-<NUM>, a synchronization accuracy of <NUM>-<NUM> is preferred.

Claim 1:
A method for tracking moving objects (<NUM>) in an environment (<NUM>) of an event, each moving object (<NUM>) comprising an associated identity marking (<NUM>) and different key portions (<NUM>), wherein the key portion (<NUM>) being used to determine which moving object (<NUM>) is first to finish, such as the nose of a horse or the chest of an athlete, said method being performed by a controller (<NUM>) and comprising:
receiving (S100) images from a camera (<NUM>) having a field-of-view in a section of the environment (<NUM>),
identifying (S102) moving objects (<NUM>) in the received images and separate these moving objects from a background of the environment,
determining (S104) position coordinates of the associated identity markings (<NUM>) and the different key portions (<NUM>) of each moving object (<NUM>),
receiving (S106) a first set of data points representing position coordinates in a field of view of a first Light Detection And Ranging, LIDAR, device (<NUM>),
receiving (S108) a second set of data points representing position coordinates in a field of view of a second LIDAR device (<NUM>), the field of view of the second LIDAR device (<NUM>) is partly overlapping the field of view of the first LIDAR device (<NUM>),
separating (S110) data points representing the moving objects (<NUM>) from data points representing a background for each of the first and the second set of data points,
modelling (S111) the position of each moving object (<NUM>) as a two-dimensional probabilistic distribution,
calculating (S112) a weighted mean value for the position coordinates of the moving objects (<NUM>) based on the first and second set of data points giving more weight to the set of data points that represents moving objects closest to one of the first and second LIDAR device (<NUM>; <NUM>), wherein the weighted mean is calculated as inversely proportional to the variances of the moving objects (<NUM>), and
reconstructing (S114) the position in the environment (<NUM>) and the associated identity marking (<NUM>) for the moving objects in a particular point of time.