TARGET TRACKING METHOD AND APPARATUS

A target tracking apparatus includes an image sensor, a controller, an image signal processor, and an event processor. A processing capability of the image signal processor is different from a processing capability of the event processor. The image sensor is configured to: capture a first raw (RAW) image, and send the first raw image to the controller. The controller is configured to: determine a motion distance, in two adjacent frames of raw images, of a target object in the first raw image, where the first raw image is a next frame of image of the two adjacent frames of raw images; determine a target processor from the image signal processor and the event processor based on the motion distance, where a processing capability of the target processor matches the motion distance; and perform target tracking on the target object based on the first raw image by using the target processor.

TECHNICAL FIELD

Embodiments of this application relate to the field of electronic technologies, and in particular, to a target tracking method and apparatus.

BACKGROUND

With development of a human-computer interaction technology, an eye tracking technology is applied increasingly widely in the field of human-computer interaction. The eye tracking technology is a technology for locating a line of sight of a human eye, integrates knowledge of discipline fields such as machinery, electronics, and optics, and has a good application prospect in fields such as artificial intelligence and machine vision, for example, used for fatigue monitoring, virtual reality, and communication auxiliary tools.

An existing eye tracking apparatus captures a raw image by using an image sensor (image sensor), where the raw image includes an image of an eyeball of a user; converts the raw image to a red, green, and blue (RGB) image by using an image signal processor (ISP); performs feature extraction on the RGB image by using a central processing unit (CPU), to obtain feature information of the RGB image, and calculates an elliptic equation of the eyeball based on the feature information; and performs eye tracking on the eyeball based on an elliptic equation calculated for each frame of a plurality of consecutive frames of raw images.

However, each frame of raw image captured by the image sensor in the existing eye tracking apparatus needs to be transmitted to the image signal processor for processing, and calculation complexity of the image signal processor is large. Therefore, computing resource consumption is high. In addition, in some cases, a target may not move or a motion degree is small. Therefore, power consumption is high. In conclusion, overall performance of the existing eye tracking apparatus is poor.

SUMMARY

This application provides a target tracking method and apparatus, to improve overall performance of the target tracking apparatus. To achieve the foregoing objective, the following technical solutions are used in this application.

According to a first aspect, this application provides a target tracking apparatus. The apparatus may include an image sensor, a controller, an image signal processor, and an event processor. A processing capability of the image signal processor is different from a processing capability of the event processor. The image sensor is configured to: capture a first raw image, and send the first raw image to the controller. The controller is configured to: determine a motion distance, in two adjacent frames of raw images, of a target object in the first raw image, where the first raw image is a next frame of image of the two adjacent frames of raw images; determine a target processor from the image signal processor and the event processor based on the motion distance, where a processing capability of the target processor matches the motion distance; and perform target tracking on the target object based on the first raw image by using the target processor.

It should be noted that both the image signal processor and the event processor in this application may be dedicated processors. The image signal processor is a processor with a high processing capability, has a complex function and high processing precision, and can perform processing with high computing complexity on an image, for example, domain format conversion, white balance correction, and gamma correction. However, power consumption is high, and a computing resource is wasted. The event processor is a processor with a low processing capability, has a simple function, saves computing resources, has low power consumption, and can perform less complex processing on an image, for example, extracting an event data stream, but has low processing precision.

According to the target tracking apparatus provided in this application, the controller may flexibly select, based on the motion distance, a target processor whose processing capability matches the motion distance, to perform target tracking on the target object, that is, determine, based on the motion distance, whether a processor with a high processing capability needs to be used to perform target tracking to improve calculation precision, or a processor with a low processing capability may be used to perform target tracking to save computing resources and reduce power consumption, to improve overall performance of the target tracking apparatus.

In addition, the target tracking apparatus provided in this application needs only one image sensor to capture a raw image, and different dedicated processors are selected to process the raw image captured by the image sensor, to implement conversion from a raw domain to an RGB domain or conversion from a raw domain to an event data stream, so that costs and overheads of the sensor can be reduced.

In an embodiment, if the target processor is the image signal processor, the image signal processor is configured to: convert the first raw image to an RGB image, and send the RGB image to the controller; or if the target processor is the event processor, the event processor is configured to: convert the first raw image to an event data stream, and send the event data stream to the controller, where the event data stream indicates a brightness change status of a pixel in the first raw image. The controller is configured to perform target tracking on the target object based on the RGB image or the event data stream.

It should be further noted that the event data stream in this application describes a brightness change status of a pixel in the first raw image. The event data stream includes a plurality of pieces of event data, each of the plurality of pieces of event data describes an event, and a brightness change of any pixel in the first raw image is referred to as an event.

In an embodiment, the controller is configured to determine the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold.

In an embodiment, the processing capability of the image signal processor is higher than the processing capability of the event processor, and the controller is configured to: if the motion distance is greater than the distance threshold, determine the image signal processor as the target processor; or if the motion distance is less than or equal to the distance threshold, determine the event processor as the target processor.

According to the target tracking apparatus provided in this application, when the motion distance of the target object in the two adjacent frames is large, the image signal processor with higher calculation precision may be matched and used to process the first raw image. On the contrary, when the motion distance of the target object in the two adjacent frames is small or the target object does not move, the event processor with low power consumption and simple calculation may be matched and used to process the first raw image. This helps balance allocation of computing resources and power consumption, to improve overall performance of a system.

In an embodiment, the controller is configured to determine first location information based on the first raw image by using the target processor, where the first location information indicates a location of the target object at a capturing moment of the first raw image.

In an embodiment, the controller is configured to: determine second location information based on a second raw image, where the second raw image is a former frame of raw image in the two adjacent frames of raw images, and the second location information indicates a location of the target object at a capturing moment of the second raw image; determine third location information based on a third raw image, where the third raw image is a latter frame of raw image in the two adjacent frames of raw images, and the third location information indicates a location of the target object at a capturing moment of the third raw image; and determine the motion distance based on the second location information and the third location information.

According to the target tracking apparatus provided in this application, based on a real-time motion status of the target object in previous two frames, a processor whose processing capability matches the motion status is selected to process a current frame of raw image, so that precision is higher.

In an embodiment, the target object includes an eyeball.

According to a second aspect, this application further provides a target tracking method. The method may include: capturing a first raw image; determining a motion distance, in two adjacent frames of raw images, of a target object in the first raw image, where the first raw image is a next frame of image of the two adjacent frames of raw images; determining a target processor from the image signal processor and the event processor based on the motion distance, where a processing capability of the image signal processor is different from a processing capability of the event processor, and a processing capability of the target processor matches the motion distance; and performing target tracking on the target object based on the first raw image by using the target processor.

In an embodiment, the performing target tracking on the target object based on the first raw image by using the target processor includes: if the target processor is the image signal processor, converting the first raw image to an RGB image by using the image signal processor; or if the target processor is the event processor, converting the first raw image to an event data stream by using the event processor, where the event data stream indicates a brightness change status of a pixel in the first raw image; and performing target tracking on the target object based on the RGB image or the event data stream.

In an embodiment, the determining a target processor from the image signal processor and the event processor based on the motion distance includes: determining the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold.

In an embodiment, the processing capability of the image signal processor is higher than the processing capability of the event processor. The determining the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold includes: if the motion distance is greater than the distance threshold, determining the image signal processor as the target processor; or if the motion distance is less than or equal to the distance threshold, determining the event processor as the target processor.

In an embodiment, the performing target tracking on the target object based on the first raw image by using the target processor includes: determining first location information based on the first raw image by using the target processor, where the first location information indicates a location of the target object at a capturing moment of the first raw image.

In an embodiment, the determining a motion distance, in two adjacent frames of raw images, of a target object in the first raw image includes: determining second location information based on a second raw image, where the second raw image is a former frame of raw image in the two adjacent frames of raw images, and the second location information indicates a location of the target object at a capturing moment of the second raw image; determining third location information based on a third raw image, where the third raw image is a latter frame of raw image in the two adjacent frames of raw images, and the third location information indicates a location of the target object at a capturing moment of the third raw image; and determining the motion distance based on the second location information and the third location information.

In an embodiment, the target object includes an eyeball.

According to a third aspect, this application further provides a control method. The method may include: obtaining a first raw image; determining a motion distance, in two adjacent frames of raw images, of a target object in the first raw image, where the first raw image is a next frame of image of the two adjacent frames of raw images; determining a target processor from the image signal processor and the event processor based on the motion distance, where a processing capability of the image signal processor is different from a processing capability of the event processor, and a processing capability of the target processor matches the motion distance; and performing target tracking on the target object based on the first raw image by using the target processor.

In an embodiment, the performing target tracking on the target object based on the first raw image by using the target processor includes: if the target processor is the image signal processor, converting the first raw image to an RGB image by using the image signal processor; or if the target processor is the event processor, converting the first raw image to an event data stream by using the event processor, where the event data stream indicates a brightness change status of a pixel in the first raw image; and performing target tracking on the target object based on the RGB image or the event data stream.

In an embodiment, the determining a target processor from the image signal processor and the event processor based on the motion distance includes: determining the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold.

In an embodiment, the processing capability of the image signal processor is higher than the processing capability of the event processor. The determining the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold includes: if the motion distance is greater than the distance threshold, determining the image signal processor as the target processor; or if the motion distance is less than or equal to the distance threshold, determining the event processor as the target processor.

In an embodiment, the performing target tracking on the target object based on the first raw image by using the target processor includes: determining first location information based on the first raw image by using the target processor, where the first location information indicates a location of the target object at a capturing moment of the first raw image.

In an embodiment, the determining a motion distance, in two adjacent frames of raw images, of a target object in the first raw image includes: determining second location information based on a second raw image, where the second raw image is a former frame of raw image in the two adjacent frames of raw images, and the second location information indicates a location of the target object at a capturing moment of the second raw image; determining third location information based on a third raw image, where the third raw image is a latter frame of raw image in the two adjacent frames of raw images, and the third location information indicates a location of the target object at a capturing moment of the third raw image; and determining the motion distance based on the second location information and the third location information.

In an embodiment, the target object includes an eyeball.

According to a fourth aspect, this application further provides a controller. The controller may include units configured to implement the method according to any one of the third aspect and the possible implementations of the third aspect.

According to a fifth aspect, this application further provides a controller. The controller may include a processor and a communication interface, the processor is coupled to the communication interface, the communication interface is configured to provide data and/or information for the processor, and the processor is configured to perform the method according to any one of the third aspect and the possible implementations of the third aspect.

According to a sixth aspect, this application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when being executed by at least one processor, the computer program is used to implement the method in any one of the first aspect and the possible implementations of the first aspect.

According to a seventh aspect, this application further provides a computer program product. When being executed by at least one processor, the computer program product is used to implement the method in any one of the first aspect and the possible implementations of the first aspect.

The target tracking method, the control method, the controller, the computer storage medium, and the computer program product provided in this application are all used to perform the target tracking method provided above. Therefore, for beneficial effects that can be achieved by the target tracking method, the control method, the controller, the computer storage medium, and the computer program product, refer to the beneficial effects in the target tracking method provided above. Details are not described herein again.

DESCRIPTION OF EMBODIMENTS

In embodiments of this application, “first” or “second” and similar words do not indicate any order, quantity, or importance, but are merely used to distinguish between different parts. Likewise, terms such as “a/an” and “one” do not indicate a quantity limitation, but mean at least one. “Coupling” and similar terms are not limited to a direct physical or mechanical connection, but may include an electrical connection. Regardless of direct or indirect, “coupling” is equivalent to a connection in a broad sense.

The term “example” or “for example” in embodiments of this application means “used as an example, an illustration, or a description”. Any embodiment or design scheme described as “example” or “for example” in embodiments of this application should not be explained as being more preferred or having more advantages than another embodiment or design scheme. To be precise, use of the word like “example” or “for example” is intended to present a relative concept in a specific manner. In the descriptions of embodiments of this application, unless otherwise stated, “a plurality of” means two or more than two. For example, a plurality of processors mean at least two processors.

An existing head mounted device includes an eye tracking apparatus for determining a gaze direction of a user. The eye tracking apparatus captures a raw image by using an image sensor, where the raw image includes an image of an eyeball of a user; converts the raw image to an RGB image by using an ISP; performs feature extraction on the RGB image by using a CPU, to obtain feature information of the RGB image, and obtains an elliptic equation of the eyeball through calculation based on the feature information; and performs eye tracking on the eyeball based on an elliptic equation calculated for each frame of a plurality of consecutive frames of raw images.

However, each frame of raw image captured by the image sensor in the existing eye tracking apparatus needs to be transmitted to the image signal processor for processing, and calculation complexity of the image signal processor is large. Therefore, computing resource consumption is high. In addition, in some cases, a target may not move or a motion degree is small. Therefore, power consumption is high.

In addition, due to limitations such as a bandwidth between the image sensor and the ISP and a processing speed of an eye tracking algorithm performed by the CPU, it is difficult to achieve a very high system frame rate (for example, higher than 240 Hz). Even if a high frame rate can be achieved, computing resource consumption is usually high. Therefore, power consumption is high.

In conclusion, overall performance of the existing eye tracking apparatus is poor.

To resolve the foregoing problem, embodiments of this application provide a target tracking method and apparatus, to improve overall performance of the target tracking apparatus.

The following first describes a target tracking apparatus provided in embodiments of this application.

In an embodiment, the target tracking apparatus provided in this application may be used in a terminal. The terminal may be a transportation tool or an intelligent device. For example, the terminal may be a vehicle (such as a self-driving vehicle, an intelligent vehicle, an electric vehicle, or a digital vehicle), an uncrewed aerial vehicle, a track vehicle, a bicycle, or a traffic light. For another example, the terminal may be a mobile phone, a tablet computer, a notebook computer, a personal digital assistant, a sales terminal, a vehicle-mounted computer, an augmented reality (AR) device, a virtual reality (VR) device, a wearable device, or a vehicle-mounted terminal.

In an embodiment, the target tracking apparatus may perform target tracking on a plurality of target objects. This is not limited in this application.

In an embodiment, the target object may include an eyeball of a user, and the target tracking apparatus may perform eye tracking on the eyeball.

In an embodiment, the target tracking apparatus provided in this application may be applicable to various target tracking application scenarios. It should be noted that the target tracking apparatus may be used in an application scenario of target tracking, and certainly may also be used in an application scenario of target detection. This is not limited in this application.

For example,FIG.1is a diagram of an application scenario of a target tracking apparatus according to an embodiment of this application. As shown in (a) inFIG.1, the target tracking apparatus may be used in a VR device. The VR device may perform eye tracking on an eyeball (for example, a pupil) of a user by using the target tracking apparatus, to control a VR game. As shown in (b) inFIG.1, the target tracking apparatus may be used in a display. The display may perform eye tracking on an eyeball (for example, a pupil) of a user by using the target tracking apparatus, to identify an area of concern of the user. As shown in (c) inFIG.1, the target tracking apparatus may be used in an intelligent vehicle. The intelligent vehicle may perform eye tracking on an eyeball (for example, a pupil) of a vehicle owner by using the target tracking apparatus, to detect whether the driver is tired while driving.

FIG.2is a diagram of a structure of a target tracking apparatus100according to an embodiment of this application. As shown inFIG.2, the apparatus100may include an image sensor110, a controller120, an image signal processor130, and an event processor140. A processing capability of the image signal processor130is different from a processing capability of the event processor140. The image sensor110is configured to: capture a first raw image, and send the first raw image to the controller. The controller120is configured to: determine a motion distance, in two adjacent frames of raw images, of a target object in the first raw image, where the first raw image is a next frame of image of the two adjacent frames of raw images; determine a target processor from the image signal processor130and the event processor140based on the motion distance, where a processing capability of the target processor matches the motion distance; and perform target tracking on the target object based on the first raw image by using the target processor.

For example, the image sensor110is a complementary metal-oxide-semiconductor (CMOS) image sensor.

It should be noted that the raw image in this application is an “unprocessed” image, is raw data of a digital signal converted by the image sensor150from a captured light source signal, and records raw information of the image sensor. In an embodiment, the first raw image may further record some metadata generated during photographing, for example, a sensitivity setting, a shutter speed, an aperture value, and white balance.

In an embodiment, the controller120is configured to determine the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold.

In an embodiment, the controller120is configured to: if the motion distance is greater than the distance threshold, determine the image signal processor as the target processor; or if the motion distance is less than or equal to the distance threshold, determine the event processor as the target processor.

It should be noted that both the image signal processor130and the event processor140in this application may be dedicated processors. The image signal processor130is a processor with a high processing capability, has a complex function and high processing precision, and can perform processing with high computing complexity on an image, for example, domain format conversion, white balance correction, and gamma correction. However, power consumption is high, and a computing resource is wasted. The event processor140is a processor with a low processing capability, has a simple function, saves computing resources, has low power consumption, and can perform less complex processing on an image, for example, extracting an event data stream, but has low processing precision.

It should be further noted that the event data stream in this application describes a brightness change status of a pixel in the first raw image. The event data stream includes a plurality of pieces of event data, each of the plurality of pieces of event data describes an event, and a brightness change of any pixel in the first raw image is referred to as an event.

For example, event data is usually represented as (x, y, t, p), where (x, y) is coordinates of a pixel at which an event occurs, t is a moment at which the event occurs, and p is a polarity of the event (for example, p=0 indicates that brightness of the pixel is reduced compared with that during previous sampling, and p=1 indicates that brightness of the pixel is increased compared with that during the previous sampling).

In an embodiment, the target tracking in this application may include determining location information of a target and/or determining movement trajectory information of the target.

In an embodiment, the controller120is configured to determine first location information based on the first raw image by using the target processor, where the first location information indicates a location of the target object at a capturing moment of the first raw image.

It should be noted that the motion distance in this application may be understood as a distance at which the target object moves in the two adjacent frames of raw images, that is, a location offset of the target object in the two adjacent frames of raw images.

In an embodiment, the controller120may determine the motion distance in a plurality of manners. This is not limited in this embodiment of this application.

In an embodiment, the controller120may receive motion information from another apparatus, where the motion information indicates the motion distance.

In another embodiment, the controller120may determine the motion distance based on the two adjacent frames of raw images.

For example, the controller120is configured to: determine second location information based on a second raw image, where the second raw image is a former frame of raw image in the two adjacent frames of raw images, and the second location information indicates a location of the target object at a capturing moment of the second raw image; determine third location information based on a third raw image, where the third raw image is a latter frame of raw image in the two adjacent frames of raw images, and the third location information indicates a location of the target object at a capturing moment of the third raw image; and determine the motion distance based on the second location information and the third location information.

In an embodiment, if the target processor is the image signal processor130, the image signal processor130is configured to: convert the first raw image to an RGB image, and send the RGB image to the controller120; or if the target processor is the event processor140, the event processor140is configured to: convert the first raw image to an event data stream, and send the event data stream to the controller120, where the event data stream indicates a brightness change status of a pixel in the first raw image. Correspondingly, the controller120is configured to perform target tracking on the target object based on the RGB image or the event data stream.

It should be noted that a process in which the image signal processor130converts the first raw image to the RGB image (that is, domain format conversion) and a process in which the event processor140converts the first raw image to the event data stream (that is, dynamic information extraction) are not limited in this application. For details, refer to the conventional technology.

For example, the target object is an eyeball. The controller120is configured to: perform feature extraction on the RGB image to obtain information about an edge point of at least one object in the RGB image (the information about the edge point may include a location of the edge point in the RGB image), where the at least one object includes the eyeball; determine information about an edge point of the eyeball based on a geometric feature (for example, a geometric relationship between an eye corner and an orbit) of the eyeball and the information about the edge point of the at least one object; and determine a first elliptic equation based on the information about the edge point of the eyeball and a second elliptic equation, where the first elliptic equation indicates a location of the eyeball at a capturing moment of the first raw image, the second elliptic equation is determined based on a third raw image, the second elliptic equation indicates a location of the eyeball at a capturing moment of the third raw image, and the third raw image is a previous frame of raw image of the first raw image.

It should be noted that, for a specific method for performing eye tracking on an eyeball in this application, refer to the conventional technology, for example, performing eye tracking based on a location of a through hole on the eyeball. This is not limited in this application.

For example, the target object is an eyeball, and first event data in the event data stream is (x1, y1, t1, p1). The controller120is configured to: calculate a distance k1between a pixel (x1, y1) and a second elliptic equation f (x, y)=(x−m1)2/a12+(y−n1)2/b2−1=0, where the second elliptic equation indicates the location of the eyeball at the capturing moment of the third raw image, and the third raw image is a previous frame of raw image of the first raw image; determine, based on the distance k1and a preset distance threshold k0, whether the pixel (x1, y1) is an edge point of the eyeball, and so on, to obtain the information about the edge point of the eyeball; and determine a first elliptic equation based on the information about the edge point of the eyeball and the second elliptic equation, where the first elliptic equation indicates the location of the eyeball at the capturing moment of the first raw image.

Optionally, a specific form of each component in the apparatus100is not limited in this application.

In an embodiment, the image sensor110, the controller120, and the event processor140may be integrated together, for example, integrated in a chip1. The image signal processor130is an independent chip, for example, a chip2.

In another embodiment, the controller120, the image signal processor130, and the event processor140may be integrated together, for example, integrated in a chip3. The image sensor110is an independent chip, for example, a chip4.

According to the target tracking apparatus provided in this application, the controller may flexibly select, based on the motion distance, a processor whose processing capability matches the motion distance, to perform target tracking, that is, determine, based on the motion distance, whether a processor with a high processing capability needs to be used to perform target tracking to improve calculation precision, or a processor with a low processing capability may be used to perform target tracking to save computing resources and reduce power consumption, to improve overall performance of the target tracking apparatus.

In addition, the target tracking apparatus provided in this application needs only one image sensor to capture a raw image, and different dedicated processors are selected to process the raw image captured by the image sensor, to implement conversion from a raw domain to an RGB domain or conversion from a raw domain to an event data stream, so that costs and overheads of the sensor can be reduced.

The following further describes a target tracking method provided in embodiments of this application.

FIG.3is a schematic flowchart of a target tracking method200according to an embodiment of this application. As shown inFIG.3, the method200may include the following operations. It should be noted that the following operations may be performed in various sequences and/or simultaneously, and are not limited to the execution sequence shown inFIG.3.

It should be noted that the following describes the method200by using an example in which the method200is applied to the image tracking apparatus100shown inFIG.2. However, this application is not limited thereto. In an embodiment, the method200may be applied to any apparatus that can implement the following operations.

In operation S201, an image sensor captures a first raw image.

In operation S202, the image sensor sends the first raw image to a controller. Correspondingly, the controller receives the first raw image from the image sensor.

In an embodiment, operations S201and S202describe a manner in which the controller obtains the first raw image. However, this application is not limited thereto. The controller may alternatively obtain the first raw image in another manner. This is not limited in this application.

In operation S203, the controller determines a motion distance, in two adjacent frames of raw images, of a target object in the first raw image, where the first raw image is a next frame of image of the two adjacent frames of raw images.

In an embodiment, the target object may include an eyeball.

In an embodiment, before determining the motion distance, in the two adjacent frames of raw images, of the target object in the first raw image, the controller may determine second location information based on a second raw image, where the second raw image is a former frame of raw image in the two adjacent frames of raw images, and the second location information indicates a location of the target object at a capturing moment of the second raw image; and determine third location information based on a third raw image, where the third raw image is a latter frame of raw image in the two adjacent frames of raw images, and the third location information indicates a location of the target object at a capturing moment of the third raw image. Correspondingly, the controller may determine the motion distance based on the second location information and the third location information.

In operation S204, the controller determines a target processor from the image signal processor and the event processor based on the motion distance, where a processing capability of the image signal processor is different from a processing capability of the event processor, and a processing capability of the target processor matches the motion distance.

Specifically, the controller may determine the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold.

In an embodiment, the processing capability of the image signal processor is higher than the processing capability of the event processor. Operation S204may include, if the motion distance is greater than the distance threshold, the controller determines the image signal processor as the target processor; or if the motion distance is less than or equal to the distance threshold, the controller determines the event processor as the target processor.

In operation S205, the controller performs target tracking on the target object based on the first raw image by using the target processor.

In an embodiment, if the target processor is the image signal processor, operation S205may include operations S206to S209.

In operation S206, the controller sends the first raw image to the image signal processor. Correspondingly, the image signal processor receives the first raw image from the controller.

In operation S207, the image signal processor converts the first raw image to an RGB image.

In operation S208, the image signal processor sends the RGB image to the controller. Correspondingly, the controller receives the RGB image from the image signal processor.

In operation S209, the controller performs target tracking on the target object based on the RGB image.

In another possible implementation, if the target processor is the event processor, S205may include operations S210to S213.

In operation S210, the controller sends the first raw image to the event processor. Correspondingly, the event processor receives the first raw image from the controller.

In operation S211, the event processor converts the first raw image to an event data stream, where the event data stream indicates a brightness change status of a pixel in the first raw image.

In operation S212, the event processor sends the event data stream to the controller. Correspondingly, the controller receives the event data stream from the event processor.

In operation S213, the controller performs target tracking on the target object based on the event data stream.

In an embodiment, that the controller performs target tracking on the target object based on the RGB image or the event data stream may include: determining first location information of the target object based on the RGB image or the event data stream, where the first location information indicates a location of the target object at a capturing moment of the first raw image.

It should be noted that, for operations that are not described in detail in the method200, refer to descriptions of related parts in the apparatus100. To avoid repetition, details are not described herein.

With reference to the target tracking apparatus shown inFIG.2, the following describes a schematic flowchart of a target tracking method according to an embodiment of this application. It should be noted that the following operations may be performed in various sequences and/or simultaneously, and an execution sequence of the operations should not be limited to a sequence indicated by sequence numbers of the operations. The procedure includes the following operations:

(1) The image sensor captures a raw image1, where the raw image1includes an eyeball object, and the eyeball object includes a pupil.

(2) The image sensor sends the raw image1to the controller.

(3) The controller sends the raw image1to the image signal processor based on preset control information1. Correspondingly, the image signal processor receives the raw image1from the controller.

(4) The image signal processor converts the raw image1to an RGB image1.

(5) The image signal processor sends the RGB image1to the controller. Correspondingly, the controller receives the RGB image1from the image signal processor.

(6) The controller performs eye tracking on the eyeball based on the RGB image1, to obtain location information1, where the location information1indicates a location of the eyeball at a capturing moment of the raw image1.

(7) The image sensor captures a raw image2, where the raw image2includes the eyeball object, and the raw image2is a next frame of image of the raw image1.

(8) The image sensor sends the raw image2to the controller.

(9) The controller sends the raw image2to the image signal processor based on the control information1. Correspondingly, the image signal processor receives the raw image2from the controller.

(10) The image signal processor converts the raw image2to an RGB image2.

(11) The image signal processor sends the RGB image2to the controller. Correspondingly, the controller receives the RGB image2from the image signal processor.

(12) The controller performs eye tracking on the eyeball based on the RGB image2, to obtain location information2, where the location information2indicates a location of the eyeball at a capturing moment of the raw image2.

(13) The image sensor captures a raw image3, where the raw image3includes the eyeball object, and the raw image3is a next frame of image of the raw image2.

(14) The image sensor sends the raw image3to the controller.

(15) The controller determines, based on the location information1and the location information2, a motion distance of the eyeball from the sampling moment of the raw image1to the sampling moment of the raw image2.

(16) The controller determines, based on the motion distance and a preset distance threshold, a target processor from the image signal processor and the event processor, where a processing capability of the target processor matches the motion distance.

In an embodiment, if the motion distance is greater than the distance threshold, the controller determines the image signal processor as the target processor, and continues to perform operations (17) to (20); or if the motion distance is less than or equal to the distance threshold, the controller determines the event processor as the target processor, and continues to perform operations (21) to (24).

(17) The controller sends the raw image3to the event processor. Correspondingly, the event processor receives the raw image3from the controller.

(18) The event processor converts the raw image3to an event data stream.

(19) The event processor sends the event data stream to the controller. Correspondingly, the controller receives the event data stream from the event processor.

(20) The controller performs eye tracking on the eyeball based on the event data flow.

(21) The controller sends the raw image3to the image signal processor.

Correspondingly, the image signal processor receives the raw image3from the controller.

(22) The image signal processor converts the raw image3to an RGB image3.

(23) The image signal processor sends the RGB image3to the controller.

Correspondingly, the controller receives the RGB image3from the image signal processor.

(24) The controller performs eye tracking on the eyeball based on the RGB image3.

In an embodiment, the controller may perform target tracking on the target object based on each frame of raw image after the raw image3by using a procedure similar to (15) to (24). This is not limited in this embodiment of this application.

The foregoing describes the target tracking method provided in embodiments of this application. The following further describes the controller in the target tracking apparatus.

FIG.4is a schematic block diagram of a controller300according to an embodiment of this application. As shown inFIG.4, the controller300may be used in the target tracking apparatus100, and may be a controller in the target tracking apparatus100. The controller300may include an obtaining unit301, a determining unit302, and a tracking unit303.

The obtaining unit301is configured to obtain a first raw image. The determining unit302is configured to: determine a motion distance, in two adjacent frames of raw images, of a target object in the first raw image, where the first raw image is a next frame of image of the two adjacent frames of raw images; and determine a target processor from the image signal processor and the event processor based on the motion distance, where a processing capability of the image signal processor is different from a processing capability of the event processor, and a processing capability of the target processor matches the motion distance. The tracking unit303is configured to perform target tracking on the target object based on the first raw image by using the target processor.

In an embodiment, the determining unit302is configured to determine the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold.

In an embodiment, the processing capability of the image signal processor is higher than the processing capability of the event processor, and the determining unit302is configured to: if the motion distance is greater than the distance threshold, determine the image signal processor as the target processor; or if the motion distance is less than or equal to the distance threshold, determine the event processor as the target processor.

In an embodiment, if the target processor is the image signal processor, the tracking unit303is configured to convert the first raw image to an RGB image by using the image signal processor; or if the target processor is the event processor, the tracking unit303is configured to convert the first raw image to an event data stream by using the event processor, where the event data stream indicates a brightness change status of a pixel in the first raw image. The tracking unit303is configured to perform target tracking on the target object based on the RGB image or the event data stream.

In an embodiment, the tracking unit303is configured to determine first location information based on the first raw image by using the target processor, where the first location information indicates a location of the target object at a capturing moment of the first raw image.

In an embodiment, the determining unit302is configured to: determine second location information based on a second raw image, where the second raw image is a former frame of raw image in the two adjacent frames of raw images, and the second location information indicates a location of the target object at a capturing moment of the second raw image; determine third location information based on a third raw image, where the third raw image is a latter frame of raw image in the two adjacent frames of raw images, and the third location information indicates a location of the target object at a capturing moment of the third raw image; and determine the motion distance based on the second location information and the third location information.

One or more of the units in the embodiment shown inFIG.4may be implemented by using software, hardware, firmware, or a combination thereof. The software or firmware includes but is not limited to computer program instructions or code, and may be executed by a hardware processor. The hardware includes but is not limited to various integrated circuits, such as a central processing unit (CPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

FIG.5is a schematic block diagram of a controller400according to an embodiment of this application. The controller400may include a processor401and a communication interface402. The processor401is coupled to the communication interface402.

The processor401in this embodiment of this application may include one or more processing units. Optionally, the processing unit includes but is not limited to a CPU, a general purpose processor, a DSP, an ASIC, an FPGA, a discrete gate, a transistor logic device, a discrete hardware component, or the like. The general-purpose processor may be a microprocessor, a microcontroller, any conventional processor, or the like.

For example, the processor401is configured to: obtain a first raw image through the communication interface; determine a motion distance, in two adjacent frames of raw images, of a target object in the first raw image, where the first raw image is a next frame of image of the two adjacent frames of raw images; determine a target processor from the image signal processor and the event processor based on the motion distance, where a processing capability of the image signal processor is different from a processing capability of the event processor, and a processing capability of the target processor matches the motion distance; and perform target tracking on the target object based on the first raw image by using the target processor.

In an optional example, a person skilled in the art may understand that the controller400may be the controller in the foregoing embodiment of the method200, and the controller400may be configured to perform procedures and/or operations corresponding to the controller in the foregoing embodiment of the method200. To avoid repetition, details are not described herein again.

Optionally, the controller400may further include a memory403.

Specifically, the memory403is configured to store program code and instructions of the target tracking apparatus. Optionally, the memory403is further configured to store information and/or data obtained in a process in which the processor401performs the foregoing embodiment of the method200, for example, a raw image and location information of the target object.

Optionally, the memory403may be an independent device or may be integrated into the processor401.

It should be noted thatFIG.5shows only a simplified design of the controller400. In actual application, the controller400may further include other necessary elements, including but not limited to any quantity of communication interfaces, processors, selectors, memories, and the like. All controllers400that can implement this application fall within the protection scope of this application.

In a possible design, the controller400may be a chip. Optionally, the chip may further include one or more memories, configured to store computer-executable instructions. When the chip apparatus runs, the processor may execute the computer-executable instructions stored in the memory, so that the chip performs the foregoing target tracking method.

Optionally, the chip apparatus may be a field programmable gate array, a dedicated integrated chip, a system chip, a central processing unit, a network processor, a digital signal processing circuit, a microcontroller, a programmable controller, or another integrated chip for implementing related functions.

An embodiment of this application further provides a computer-readable storage medium. The computer-readable storage medium stores computer instructions. When the computer instructions are run on a computer, the target tracking method described in the foregoing method embodiment is implemented.

An embodiment of this application further provides a computer program product. When the computer program product runs on a processor, the target tracking method described in the foregoing method embodiment is implemented.

The target tracking apparatus, the computer-readable storage medium, the computer program product, or the chip provided in embodiments of this application are all configured to perform the corresponding target tracking method provided above. Therefore, for beneficial effects that can be achieved by the target tracking apparatus, the computer-readable storage medium, the computer program product, or the chip, refer to the beneficial effects in the corresponding target tracking method provided above. Details are not described herein again.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, division into the modules is merely logical function division. There may be another division manner during actual implementation. For example, a plurality of modules or components may be combined or integrated into another apparatus, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces. The indirect couplings or communication connections between the apparatuses may be implemented in electronic, mechanical, or other forms.