OBJECT AND FEATURE DETECTION IN IMAGES

A processing device is configured to obtain a sequence of images of a scene captured by an image sensor, determine an analysis area for an object in a respective image in the sequence of images, and process the respective image within the analysis area for detection of predefined features of the object. The processing device is further configured to receive pose prediction data, PPD, which represents predicted poses of the object as a function of time, and to determine the analysis area based on the PPD. The PPD may be given by three-dimensional poses of the object that have been determined in the system based on images from a plurality of image sensors in the system. The PPD facilitates detection of features of individual objects in the images even if the objects are occluded and/or crowded.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Swedish Patent Application No. 2150747-0, filed Jun. 11, 2021, the content of which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to image processing for recognition and positioning and, in particular, to image processing for detection of predefined features of individual objects in image data for use in 3D pose reconstruction.

BACKGROUND ART

Feature detection of objects in images is an integral and important part of many image processing techniques. One such image processing technique is three-dimensional (3D) pose reconstruction based on two-dimensional (2D) images. The 2D images are at least partly overlapping and generated by image sensors that are arranged at different locations and/or with different orientations in relation to a scene. In a top-down approach of 3D pose reconstruction, object detection is first performed to detect and discriminate between individual objects in the 2D images, followed by feature detection for the respective object, whereupon the 3D pose of the respective object is determined by triangulation based on the locations of corresponding features of the same object in different 2D images. In a bottom-up approach, feature detection is first performed to detect predefined features irrespective of object, followed by association processing to assign detected features to objects, whereupon the 3D pose of the respective object is determined by triangulation based on the locations of corresponding features of the same object in different 2D images. Irrespective of approach, it is challenging to perform 3D pose reconstruction when objects are crowded and/or occluded in the 2D images, as well as when the 2D images include moderate or high levels of noise. There is thus a general need to improve accuracy of 3D pose reconstruction.

There is also a general need to increase the speed of 3D pose reconstruction, for example to enable real-time processing. Likewise, there is a general need to improve the power-efficiency of 3D pose reconstruction, for example to enable all or part of the 3D pose reconstruction to be performed by power-limited devices.

BRIEF SUMMARY

It is an objective to at least partly overcome one or more limitations of the prior art.

Another objective is to improve detection of features of objects in 2D images for use in 3D pose reconstruction.

Yet another objective is to enable at least one of increased processing speed and/or reduced power consumption of feature detection in 2D images when performed in accordance with the top-down approach.

One or more of these objectives, as well as further objectives that may appear from the description below, are at least partly achieved by processing devices, a system, methods, and a computer-readable medium according to the independent claims, embodiments thereof being defined by the dependent claims.

Still other objectives, as well as features, aspects and technical effects will appear from the following detailed description, from the attached claims as well as from the drawings.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Also, it will be understood that, where possible, any of the advantages, features, functions, devices, and/or operational aspects of any of the embodiments described and/or contemplated herein may be included in any of the other embodiments described and/or contemplated herein, and/or vice versa. In addition, where possible, any terms expressed in the singular form herein are meant to also include the plural form and/or vice versa, unless explicitly stated otherwise. As used herein, “at least one” shall mean “one or more” and these phrases are intended to be interchangeable. Accordingly, the terms “a” and/or “an” shall mean “at least one” or “one or more”, even though the phrase “one or more” or “at least one” is also used herein. As used herein, except where the context requires otherwise owing to express language or necessary implication, the word “comprise” or variations such as “comprises” or “comprising” is used in an inclusive sense, that is, to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments. The term “compute”, and derivatives thereof, is used in its conventional meaning and may be seen to involve performing a calculation involving one or more mathematical operations to produce a result, for example by use of a computer.

As used herein, the terms “multiple”, “plural” and “plurality” are intended to imply provision of two or more elements, whereas the term a “set” of elements is intended to imply a provision of one or more elements. The term “and/or” includes any and all combinations of one or more of the associated listed elements.

Well-known functions or constructions may not be described in detail for brevity and/or clarity. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

Like numbers refer to like elements throughout.

Before describing embodiments in more detail, a few definitions will be given.

As used herein, “scene” denotes a three-dimensional (3D) space that is collectively monitored by two or more imaging devices. The imaging devices have at least partly overlapping fields of views. The respective imaging device is configured to produce a digital video stream, i.e. a coherent time-sequence of digital images. The respective image is a two-dimensional (2D) representation of the scene, or part thereof, as seen by the imaging device. The imaging device may comprise imaging optics, a digital image sensor responsive to electromagnetic radiation, and control electronics for acquiring signals from the digital image sensor and generating a digital image, which may be monochromatic or polychromatic. The respective imaging device may be responsive to electromagnetic radiation in any wavelength range, including but not limited to ultraviolet, visible or infrared radiation, or any part or combination thereof.

As used herein, “field of view” has its conventional meaning and denotes the extent of the scene that is observed by the respective imaging device at any given moment and may be defined as a solid angle through which the imaging device is sensitive to the electromagnetic radiation.

As used herein, “keypoint” has its conventional meaning in the field of computer vision and is also known as an interest point. A keypoint is a spatial location or point in an image that define what is interesting or what stand out in the image and may be defined to be invariant to image rotation, shrinkage, translation, distortion, etc. More generally, a keypoint may be denoted a “reference point” on an object to be detected in the image, with the reference point having a predefined placement on the object. Keypoints may be defined for a specific type of object, for example a human body, a part of the human body, or an inanimate object with a known structure or configuration. In the example of a human body, keypoints may identify one or more joints and/or extremities. Keypoints may be detected by use of any existing feature detection algorithm(s), for example image processing techniques that are operable to detect one or more of edges, corners, blobs, ridges, etc. in digital images. Non-limiting examples of feature detection algorithms comprise SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Feature), FAST (Features from Accelerated Segment Test), SUSAN (Smallest Univalue Segment Assimilating Nucleus), Harris affine region detector, and ORB (Oriented FAST and Rotated BRIEF). Further information about conventional keypoint detectors is found in the article “Local invariant feature detectors: a survey”, by Tuytelaars et al, published in Found. Trends. Comput. Graph. Vis. 3(3), 177-280 (2007). Further examples of feature detection algorithms are found in the articles “Simple Baselines for Human Pose Estimation and Tracking”, by Xiao et al, published at ECCV 2018, and “Deep High-Resolution Representation Learning for Human Pose Estimation”, by Sun et al, published at CVPR 2019. Correspondingly, objects may be detected in images by use of any existing object detection algorithm(s). Non-limiting examples include various machine learning-based approaches or deep learning-based approaches, such as Viola—Jones object detection framework, SIFT, HOG (Histogram of Oriented Gradients), Region Proposals (RCNN, Fast-RCNN, Faster-RCNN), SSD (Single Shot MultiBox Detector), You Only Look Once (YOLO, YOLO9000, YOLOv3), and RefineDet (Single-Shot Refinement Neural Network for Object Detection).

As used herein, “pose” refers to a collection of positions that define the posture of an object. The pose may define the posture of the object in an image. Such a pose is denoted 2D pose and comprises a collection of 2D positions in the image. The pose may define the posture of the object in a scene. Such a pose is denoted 3D pose and comprises a collection of 3D positions in the scene.

Embodiments are related to image processing for detection of features of individual objects in 2D images. Embodiments are based on the top-down approach, described in the Background section, which involves object detection followed by feature detection. The following description will be given for human objects but is equally applicable to animals as well as inanimate objects. Embodiments will be described in relation to 3D pose reconstruction, in which the 3D pose of an object in a scene is determined based on images taken at different angles relative to the scene, and thus the object.

FIG.1Ashows an example arrangement of a monitoring system1, which may implement various embodiments. The system1is arranged to monitor a scene5in a room100. In the illustrated example, three individuals10are in the room100. The system1comprises a plurality of imaging devices2, for example digital cameras, which are oriented with their respective field of view20towards the scene5. For simplicity, the imaging devices are denoted “cameras” in the following. The scene5is associated with a fixed 3D coordinate system30(“scene coordinate system”). The cameras2may be fixed or moveable, and their relative positions and orientations are known for each image taken. The cameras2may be synchronized to capture a respective image at approximately the same time, or at least with a maximum time difference which depends on the expected maximum speed of movement of the objects10. In one example, a maximum time difference of 0.1-0.5 seconds may provide sufficient accuracy for normal human motion.

The cameras2are connected for data communication with a reconstruction device3. The data communication may be performed by wire or wirelessly, based on any standardized or proprietary protocol. The cameras2are configured to transfer image-related data to the reconstruction device3, which is configured to process the image-related data to determine the 3D pose of at least one object in the scene5. The 3D pose is given by locations of a plurality of features of the object in the scene coordinate system30. In the example ofFIG.1, the system1further comprises a monitoring device4, which is configured to receive 3D pose data from the reconstruction device3. The monitoring device4may be configured to store, analyze, process or present the 3D pose data. In one example, the monitoring device may perform so-called action or activity recognition based on time sequences of 3D poses generated by the reconstruction device3.

The system1may be seen to implement an image processing method for determining 3D poses. The method may be partitioned in different ways between the camera2and the reconstruction device3. In some embodiments, the respective camera2transfers its images, optionally pre-processed, to the reconstruction device3, which is configured to process the images from the cameras2for object detection, feature detection, and 3D pose reconstruction. Examples of such embodiments will be described further below with reference toFIG.3B. In some embodiments, the respective camera2is configured to process its images by object detection and feature detection to determine a 2D pose and to transfer 2D pose data to the reconstruction device3, which is configured to perform 3D pose reconstruction based on the 2D poses from the cameras2. Examples of such embodiments will be described further below with reference toFIG.3A.

The above-mentioned object detection and feature detection in an image results in 2D pose data, which represents the 2D pose of one or more objects in the image. The 2D pose is given by locations of a plurality of predefined features of the object in a fixed coordinate system of the image (“image coordinate system”).

The predefined features may be or include the above-mentioned keypoints. An example of keypoints K1-K14that may be detected for a human individual is shown inFIG.1C. However, any number of keypoints may be detected depending on implementation. As understood fromFIG.1C, the keypoints have a predefined location of the object and their locations in an image define the 2D pose of the object.

FIG.1Bshows an example of 2D pose data that may be generated in the system1for images taken by the three cameras2inFIG.1A. AlthoughFIG.1Bshows a table, the 2D pose data may be given in any format. In the illustrated example, the 2D pose data represents each image by a respective view identifier (1-3 inFIG.1B) and each object by a respective object identifier (1-3 inFIG.1B) and comprises a keypoint position for each keypoint detected for the respective object (L1-L14inFIG.1B). If a keypoint is not detected, the location of this keypoint may be represented by a predefined value (for example, a null value).

Object detection is performed to detect the individual objects in the image. The object detection results in an analysis area for the respective object in the image. An example of an image is given inFIG.1D, in which three analysis areas BB1, BB2, BB3have been defined, one for each object. The analysis area may be defined as a bounding box, as shown, around the object. The feature detection is then operated on the respective analysis area BB1, BB2, BB3to detect predefined features of the object, indicated as points inFIG.1D. The locations of the features are determined in the image coordinate system32.

One technical challenge in this context is to handle occlusions in which an object is partially hidden in an image, for example behind another object, and/or crowding in which objects are in close vicinity to each other in one or more views. For example, as understood fromFIG.1D, it may be challenging to define the analysis areas BB1-BB3when the objects are standing close to each other.

Another challenge may be to perform the imaging processing in a processing efficient way to save computing resources and, possibly, to enable real-time processing. Such real-time processing may, for example, allow real-time tracking of objects and 3D poses based on video streams from the cameras, for example by the monitoring device4inFIG.1A.

FIG.2Ais a flow chart of processing that may be performed in the system1. The processing comprises an image processing method200, which receives images I and outputs 2D pose data (“2DP”). The images I are generated by one of the cameras2. The method200comprises a sequence of steps that may be repeated for each incoming image. Step201comprises receiving or inputting the image. Step202comprises processing the image for detection of objects, also referred to as “object detection” herein. Each of the detected objects may be represented by an analysis area in the image (cf. BB1-BB3inFIG.1D). Step203comprises selecting one of the objects detected by step202. Step204comprises processing the selected object for detection of features, also referred to as “feature detection” herein. Step204may operate any suitable feature detection algorithm on the analysis area for the selected object, for example any of the algorithms mentioned above. Steps203-204may be repeated for all objects in the image I. Step203may be omitted if the images are known to only include one object. Step205comprises post-processing the features detected by step204for the respective object, to generate 2D pose data, 2DP, that represents a detected 2D pose of each object.

The system1may comprise one instance of the method200for each camera2. Each such instance of the method200may generate 2DP for a respective image I captured by one of the cameras2. It may be noted that the cameras2generate time sequences of images, which are processed by the instances of the method200into corresponding time sequences of 2DP.

As shown inFIG.2A, the processing in the system1also comprises a step211that performs 3D pose reconstruction, also known as “3D pose determination”. Step211comprises receiving 2DP for all objects that are detected by the different instances of the method200in time-synchronized images generated by the cameras2. Step211processes the 2DP to match 2D poses of the same object as detected by different cameras and to calculate, based on the thus-matched 2D poses, a 3D pose for each object in the scene coordinate system30(FIG.1A). The matching or association of 2D poses from different cameras may be performed by any conventional technique. The skilled person understands that a 3D position of a keypoint may be computed based on 2D positions of the keypoint in images taken by two or more cameras2, and by use of calibration data for the cameras2. The calibration data includes one or more parameters that define the positions and orientations of the cameras2(or their image sensors) in relation to each other and/or the scene. Such calibration data may, for example, be determined by calibration, as is well known in the art. Step211may operate any known triangulation algorithm on time-synchronized 2D poses that are included in 2DP for two or more cameras2. Non-limiting examples of triangulation algorithms include linear triangulation algorithms, optimal triangulation algorithms, and branch-and-bound triangulation algorithms.

The calculated 3D pose may be provided for storage, analysis, processing or presentation, for example by the monitoring device4or the reconstruction device3. However, as shown inFIG.2A, the calculated 3D pose may also be used by a step221that performs 3D pose prediction. Step221comprises receiving a time sequence of 3D poses of an object from step211and predicting the 3D pose of the object at one or more future time points. Predicting a 3D pose at future time points is equivalent to forecasting 3D motion. This is an active research field and there many available techniques for forecasting 3D motion, for example as described in “Forecasting Characteristic 3D Poses of Human Actions”, by Diller et al, published in arXiv:2011.15079v2 [cs.CV] 7 Apr. 2021, and references cited therein. Further examples are found in “Anticipating many futures: Online human motion prediction and synthesis for human-robot collaboration”, by BUtepage et al, published in arXiv:1702.08212v1 [cs.RO] 27 Feb. 2017, and references cited therein. A further example is described below with reference toFIG.9.

Step221may thus operate a 3D pose prediction algorithm on time sequences of 3D poses that are calculated by step211to predict one or more 3D poses of the respective object. Step221generates pose prediction data, PPD, which includes predicted poses of the respective object as a function of time. Depending on implementation, the PPD may be generated to include predicted 3D poses in the scene coordinate system30(denoted “3D-PPD”) or predicted 2D poses in the image coordinate system32of the respective camera2(denoted “2D-PPD”). For example, step221may project predicted 3D poses into 2D poses by use of the above-mentioned calibration data for the respective camera2. It should be noted that the predicted 2D poses will be specific to each camera2. Thus, step221may result in 2D-PPD that includes predicted 2D poses for all of the cameras2or one specific 2D-PPD for each camera2. In some embodiments, the 2D poses generated by projection of predicted 3D poses may be further processed to optimize the 2D-PPD for each camera. In one example, the projected 2D poses that are not within the field of view of a respective camera 2 may be excluded in the 2D-PPD for this camera. In another example, occlusion may be deduced within the field of view of the respective camera, and projected 2D poses that are found to be fully or heavily occluded may be excluded in the 2D-PPD for the respective camera.

As shown inFIG.2A, the PPD is received by step202, which uses the PPD when detecting the objects in the incoming images I. As will be understood from the following, the provision of the PPD has the potential of improving the object detection performed by step202, in terms of speed and/or accuracy.

FIG.2Bshows steps202A,202B that may be performed by step202to capitalize on the provision of the PPD. Step202A comprises receiving the PPD, and step202B comprises determining an analysis area for each object based on the PPD. Step202B may match the PPD to the incoming images, at least temporally, to determine an expected extent of the respective object in one or more of the incoming images and set the analysis area in relation to the expected extent. The analysis area defines a sub-region of the image and may be of any shape and may be defined to include or surround the expected extent of the object. In some embodiments, the analysis area is a bounding box similar to the boxes BB1-BB3shown inFIG.1D. In the following, the object detection performed by steps202A-202B is referred to as “predicted object detection”.

If step221generates the PPD to include predicted 2D poses (2D-PPD), the PPD is already spatially matched to the incoming images. In this context, “spatially matched” implies that the predicted poses in the PPD are given in the image coordinate system of the incoming images. On the other hand, if step221generates the PPD to include 3D poses (3D-PPD), step202B comprises projecting the 3D poses into the image coordinate system of the incoming images by use of the above-mentioned calibration data, to thereby spatially match the predicted poses in the PPD to the incoming images.

Reverting toFIG.2A, step202may also be arranged to perform conventional object detection, in which the incoming images are processed for detection of additional objects, for example by use of any of the object detection algorithms mentioned above.

In the following, such conventional object detection is referred to as “full object detection”. The full object detection does not restrict the image processing to analysis areas and is considerably more processing intensive and thus power consuming than the predicted object detection. The full object detection may be performed at startup, so that the method200generates 2DP for all objects in the scene. At startup, the full object detection is performed on images captured by at least two cameras, so that step211receives at least two 2D poses and is able to perform 3D pose determination. Thereby, the output of the full object determination is included in the PPD.

It is realized that the provision and use of the PPD will facilitate detection of features of individual objects in the incoming images, even if the objects are occluded and/or crowded in the images. It is also realized that the power consumption of the image processing method200is reduced considerably by the use of the predicted object detection.

In the example ofFIG.2A, it may also be noted that the PPD is generated based on 3D poses of the objects in the scene. Thus, the PPD represents 3D poses. This means that the predicted extent of the objects in the images is determined by taking into account the movement of the objects in 3D space. By using a time sequence of 3D poses for the prediction, it is possible to extract movement information that is not derivable from individual images and provide a high-quality prediction of the extent of the objects in the images. The use of 3D poses also makes it possible to apply realistic constraints for the movement of an object in three-dimensional space when generating the PPD. In the example of human objects, the prediction by step221may apply constraints that represent 3D movements that are impossible or unrealistic for human objects.

FIG.3Ais a block diagram of an example partitioned implementation of the processing inFIG.2Ain a system1. The cameras2are configured to perform the image processing method200and transfer 2D pose data, 2DP, to the reconstruction device3, which is configured to perform 3D pose prediction in accordance with step211. As shown, the reconstruction device3outputs one or more 3D poses (3DP), for example for receipt by the monitoring device4inFIG.1A. The reconstruction device3is also configured to perform 3D pose prediction in accordance with step221and transfer the resulting PPD to the cameras2. The partitioning inFIG.3Aprovides scalability to the system1, in that an increasing number of cameras2results in a moderate increase in processing load of the reconstruction device3. Further, the power consumption of the cameras2may be drastically reduced by the use of the predicted object detection compared to the use of full object detection only. Thereby, the partitioned implementation has the potential of increasing the service interval of battery-powered cameras2.

FIG.3Bis a block diagram of a centralized implementation of the processing inFIG.2A. The cameras2are configured to transfer the images Ito the reconstruction device3, which is a “processing device” configured to perform the image processing method200, the 3D pose prediction in accordance with step211, and the 3D pose prediction in accordance with step221. In the centralized system1, the use of the predicted object detection results in the technical advantage of increasing processing speed and/or accuracy compared to the use of conventional object detection. The increase in processing speed may, for example, be used to increase the frame rate of the cameras2.

In some implementations, whether partitioned or centralized, the reconstruction device3may be included or embodied in one of the cameras2.

FIG.4is a block diagram of a camera2and a reconstruction device3in a partitioned system. The camera2may be seen as a “processing device” with an image sensor51. The camera2is configured to perform the image processing method200inFIG.2A, and the reconstruction device3is configured to perform steps211and221inFIG.2A. In the illustrated example, the camera2comprises an object detection module52, which is configured to perform steps201-202, and feature detection module53, which is configured to perform steps203-204, and a 2D pose determination module54, which is configured to perform step205. For the avoidance of doubt, module53may be configured to selectively perform predicted object detection or full object detection, or both. Object detection in accordance with steps201-202is also abbreviated OD in the following. The communication module55is configured to establish data communication with a corresponding communication module65in the reconstruction device3on a wired or wireless communication path. As shown, module52receives an incoming image I from the image sensor51and PPD from the communication module55and outputs a set of analysis areas, [BB], for the object(s) in the image I. The module53receives and processes the image I based on [BB] and outputs location data, [L], for a plurality of keypoints in the respective analysis area. The location data defines the locations of the detected keypoints in the image coordinate system (32inFIG.1D). The module53receives [L] and outputs the 2DP for the objects in the image I. The communication module55is then operated to transfer the 2DP to the reconstruction device3. Depending on implementation, the camera may transfer 2DP for a single image or for a plurality of consecutive images.

In the example ofFIG.4, the reconstruction device3comprises a triangulation module61which is configured to perform step211, a storage module62, a prediction module63which is configured to perform step221, and a communication module65. As indicated by dashed lines, the device3may also comprise a latency estimation module64. The communication module65is configured to communicate with the communication module55. The module65receives 2DP from the camera2. As understood from the foregoing, module65also receives 2DP from one or more other cameras (not shown) and/or from a camera (not shown) in the reconstruction device3. Module61receives all the incoming 2DP and outputs 3DP for storage in module62. As understood from the foregoing, module61is configured to generate a 3D pose based on 2D poses from images captured by two or more cameras in time synchronization. Thus, 3DP may include one or more 3D poses, depending on the number of 2D poses that are transmitted in 2DP from each camera. The storage module62is configured to hold a history database, which contains the most recent 3D poses that have been determined by the module61. The history database is updated for each incoming 3DP from module61. The storage module62may comprise management function that operates on the history database to remove the oldest 3D poses as new 3D poses are added (First-In-First-Out, FIFO). The prediction module63is configured to receive a predefined number of the most recent 3D poses, arranged in time sequence, to the prediction module63. This time sequence of 3D poses is designated as [3DP] inFIG.4. The communication module65is then operated to transfer the PPD generated by module63to the camera2and to other cameras in the system. As noted above, depending on implementation, the same or different PPD may be transferred to the different cameras in the system.

Although not shown inFIG.4, the calculated 3DP may be output by the reconstruction module3, for example via the communication module65, for use in any type of post-processing or analysis. Alternatively, such post-processing or analysis may be performed by the reconstruction module3. Thus, generally, the system1may be configured to store, output or process 3DP as calculated by the reconstruction device3.

The operation of the system components inFIG.4is further illustrated in the sequence diagram ofFIG.7A. The system components perform a time sequence of round-trips, each starting with transmission of 2DP and ending with receipt of PPD. In such a round-trip, the camera2transmits 2DP at t1. The reconstruction device receives 2DP at t2, performs pose processing PP by modules61-63and transmits the resulting PPD at t3. The camera2receives PPD at t4. The time period t1-t4is denoted latency period and designated LT. After receiving the PPD, the camera2uses the PPD for object detection OD by module52(“predicted OD”). In the illustrated example, the camera2uses the PPD for object detection until the next PPD is received, at time t4′. This is further illustrated in the chart ofFIG.7B, where [3DP] is the time sequence of 3D poses that are available for the 3D pose prediction by module63during the pose processing PP, and [3DP]* is a time sequence of predicted 3D poses generated by the module63operating a pose prediction algorithm on [3DP], as indicated by arrow P inFIG.7B. The PPD is then generated by module63to represent at least part of [3DP]*.FIG.7Bindicates a prediction time window, PTW, which defines the time span of 3D poses that needs to be predicted by the module63when generating the PPD. As understood fromFIG.7A, the PTW should include the time period t4-t4′, if predicted OD is to be performed in this time period, to ensure seamless and continuous operation of predicted OD. The skilled person realizes that the time point t4′, and thus the extent of the PTW, is affected by the latency period LT, which may vary during operation of the system, for example as a result of fluctuations in the quality of the communication path or variations in the processing load of the camera2and/or the reconstruction device3. In some embodiments, PTW may be fixed and set in view of a predefined maximum latency period for the system. In other embodiments, LT may be measured or estimated in the system and PTW may be dynamically set based on the measured/estimated LT. The latency period may be estimated in any conventional way, for example based on measurements of previous roundtrip times. Alternatively or additionally, the latency period may be estimated by a neural network which has been trained to estimate or predict the latency period based on a plurality of communication-related parameters, such as previous roundtrip times, signal strength, node telemetry, etc. In the example ofFIG.4, module64is configured to provide a current value of LT to module63, which adjusts the PTW accordingly.

As understood fromFIG.7B, [3DP]* for the latency period LT is redundant since it is not used in the subsequent predicted OD. Thus, to reduce data traffic, an initial part of [3DP]* may be excluded from the PPD. In some embodiments, the extent of the initial part may be fixed and correspond a predefined minimum latency period for the system. In other embodiments, the extent of the initial part may be dynamically set based on the measured/estimated LT.

The sequence diagram inFIG.7Ais applicable both to an implementation in which the camera2continuously, at consecutive time steps, transmits an 2DP that includes 2D poses for one or more objects at the respective time step, and an implementation in which the camera 2 accumulates a time sequence of 2D poses for the object(s) and include the accumulated time sequence in the 2DP that is transmitted to the reconstruction device3. The latter implementation may serve to reduce data traffic between the camera2and the reconstruction device3.

InFIG.4, the triangulation module61and the prediction module63are physically separated from the camera2. In a variant, the modules61,63are also physically separated from each other.

FIG.5Ais a graphical illustration of PPD received and used by step202in the method200(FIG.2A). The PPD comprises a parametric representation of the predicted pose of an object as a function of time. In this context “as a function of time” merely implies that the PPD defines a timeline of predicted poses of the object, where the timeline may or may not be branched (below). In the example ofFIG.5A, the PPD defines a predicted pose (2D or 3D) at a starting time t10and at subsequent time points t11, t12, t13. There may be any time distance between the time points t10, t11, t12, t13.

The predicted poses may be seen as consecutive states 1-4 of the timeline. The parametric representation may be given in any format to represent a movement pattern of object features over time. In one example, the parametric representation comprises start locations (coordinates) of object features (for example, keypoints) at a start time, and movement parameters of the object features for one or more time periods defined in relation to the start time. Such movement parameters may define transformations in two or three dimensions, for example as indicated by arrows inFIG.5A. Examples of transformations include translation and rotation. The transformations may be represented in either a global reference frame or some local, object or part-specific reference frame. In another example, which may be particularly suitable for 2D-PPD, the parametric representation comprises locations of object features at a starting time and at subsequent time points.

FIG.5Bis a graphical illustration of PPD that defines a branched timeline of predicted poses. The different branches define possible and alternative progressions of the predicted movement of the respective object. Specifically, the PPD may define a branching time point (BTP) which separates a progressing time sequence of predicted poses into two or more alternative sequences of poses. InFIG.5B, the PPD defines a BTP at t11, at which the timeline is branched into either state 3a or state 3b at t12. Further, one BTP at t13branches the timeline from state 4a into either state 5a or state 5d, and one BTP at t13 branches the timeline from state 4b into either state 5b or state 5c. The provision of a branched timeline presumes that the pose prediction algorithm used in step212is capable of predicting alternative progressions of the movement of the object. In addition, the pose prediction algorithm may be capable of predicting a probability value of transitioning into each branch. The use of a branched timeline has the potential of improving the object detection by step202, by allowing step202to determine a predicted pose based on recently detected 2D poses, as will be described below with reference toFIG.6B.

FIG.6Ais a flow chart of an example procedure included in step202of the image processing method200. The procedure comprises the same steps202A,202B as the procedure inFIG.2B. Step202B comprises an optional step202′, which may be performed if the PPD includes 3D poses (3D-PPD). Specifically, in step202′, the PPD is processed to spatially match one or more 3D poses to the image coordinate system.

For example, step202′ may use the above-mentioned calibration data to project 3D poses into 2D poses. In a variant, step202′ may be performed as part of the step202″ (cf.FIG.6Bbelow). If the PPD includes 2D poses (2D-PPD), step202′ is omitted. In step202″, a predicted 2D pose of the object is determined based on the PPD by temporal matching. In this context, “temporally matching” implies that the predicted 2D pose is determined, at least approximately, for the time point when the image was captured. For simplicity, this time point is referred to as the “current time”. In step202′″, the analysis area is defined with respect to the predicted 2D pose from step202″, for example such that the analysis area surrounds the predicted 2D pose, optionally with a margin to account for uncertainty in the pose prediction.

FIG.6Bis a flow chart of an example procedure that may be included in step202″ inFIG.6A. The procedure inFIG.6Bis applicable to both branched and non-branched PPD. In step240, the PPD is evaluated for detection of a BTP in relation to the current time. If no BTP is detected, the procedure proceeds to step242, in which a predicted 2D pose is determined for the current time. In one example, the predicted 2D pose is extracted among a time sequence of 2D poses defined by 2D-PPD, or 3D-PPD after processing by step202′ (FIG.6A). In another example, in which 3D-PPD is used and step202′ is omitted, the predicted 2D pose is extracted by calculating a predicted 3D pose at the current time based on the parametric representation in the 3D-PPD, and by projecting the thus-predicted 3D pose onto the image by analogy with step202′. As indicated inFIG.6B, step242may be preceded by an optional interpolation step241. Step241comprises performing a temporal interpolation among predicted poses in the PPD. Step241may be applied if there is a mismatch between the current time and time points of predicted poses in the PPD. For example, if the current time point falls between time points of two predicted 2D poses in the PPD, an interpolated 2D pose for the current time may be generated based on these two predicted 2D poses, by interpolation between corresponding feature locations in these two predicted 2D poses.

If a BTP is detected by step240, the procedure proceeds to step243, in which one or more previously detected 2D poses (by step205in previous repetitions of the method200) are evaluated in relation to the predicted poses in the branches that originate at the BTP. The previously detected 2D poses are associated with time points at or near the BTP. In one example, step243comprises temporally matching each previously detected 2D pose to a predicted pose in the branches and calculating a difference between the temporally matched poses. It is realized that this evaluation needs to be performed sometime after the BTP, so that there is at least one detected 2D pose to be used in the evaluation. In another example, step243comprises operating a pose prediction algorithm on the previously detected 2D poses to generate one or more predicted 2D poses after the BTP and calculating a difference between the predicted 2D pose(s) and temporally matched poses in the different branches. In a non-limiting example, the difference between poses may be calculated as an aggregation of differences in location of corresponding keypoints between poses. In step244, one of the branches is selected based on the evaluation in step243. This branch is then used in subsequent repetitions of the method200(cf.FIG.2), although it is possible to allow steps243-244to switch between branches within a given time period after a BTP. Step244is followed by steps245and246, which correspond to steps241and242, with the only difference that steps245and246operates on the branch that was selected by step244.

In some embodiments, full object detection is performed intermittently by step202. Such intermittent activation of full object detection may be used to ensure that new objects entering the scene are properly detected and processed for feature detection, and that predicted poses of the new objects are included in the PPD for use in predicted object detection at a later time. The intermittent activation of full object detection may also be used to allow the image processing method200to recover a previously detected object that the method200, for some reason, is unable to detect for a period of time. An example implementation is shown inFIG.6C, in which step202performs the predicted object detection (step253) by default. In step251, a switching condition is evaluated to determine if full object detection (step252) should be performed for a limited time period. The full object detection252may be performed instead of or in parallel with the predicted object detection253. In the centralized system inFIG.3B, the reconstruction device3may freely switch to full object detection since it has access to images captured by cameras2in the system. In the partitioned system inFIG.3A, the cameras2may be operated jointly to perform the full object detection to enable 2D poses, which are detected by the full object detection, to be included in the PPD by the reconstruction device3. To limit the power consumption of the system (centralized or partitioned), the full object detection may be performed on images captured by a subset of the available cameras in the system. The subset may or may not vary over time and may be selected based on any suitable criterion, including but not limited to round-robin, random, statistical importance of the camera, estimation of crowdedness in the images from the camera, lost 3D poses, etc. In some embodiments, the subset is chosen so that the respective object is likely to be included in at least one image at each time instant. The camera selection may be controlled by a trigger signal from the reconstruction device3. In the partitioned system, step251may switch into, and possibly out of, the full object detection upon receipt of the trigger signal. Alternatively, step251may switch to full object detection based on a timing schedule stored in the cameras2.

For the avoidance of doubt, all embodiments described with reference toFIGS.5-6are applicable to both the partitioned system inFIG.3Aand the centralized system inFIG.3B.

The present Applicant has found the performance of step205of the method inFIG.2Amay be improved by use of the predicted 2D poses that are included in the 2D-PPD or determined from the 3D-PPD. Specifically, when determining detected 2D poses in step205, the predicted 2D poses may be used to perform a validation of the features that are detected in step204. In the validation, locations of features that are detected within an analysis area of an image may be compared to the locations of corresponding features in a predicted 2D pose that has been determined for the image. In this validation, if one or more detected feature locations are found to deviate significantly from the predicted feature locations, an error signal may be generated. The error signal may be used by the step204to modify the feature detection algorithm and/or by step221to modify the 3D pose prediction algorithm and/or by step202to modify the processing of the PPD, for example the selection of branch.

In some embodiments, the feature detection step204outputs a set of candidate locations and associated confidence values for the respective feature (keypoint). The candidate locations are different possible locations of the respective feature within the analysis area, and the confidence value designates the probability that the respective feature is located at the candidate location. For example, some feature detection algorithms generate so-called confidence maps, which are probability density functions that represent the probability of a feature being located at different locations in the analysis area. Examples of such feature detection algorithms include OpenPose and HRNet. Conventionally, the strongest peak in each confidence map is selected as the location of the respective feature, optionally after filtering of the confidence maps for removal of noise.

The left-hand side ofFIG.8Ashows an example of a confidence map CM for a keypoint corresponding to the right wrist of a human subject, with larger confidence values being represented by darker colors. The confidence map CM comprises a first region271A of elevated confidence values centered on the right wrist and a second region271B of elevated confidence values centered on the left wrist. In this example, the maximum confidence value is larger in region271B than in region271A. Thus, the feature detection algorithm in step204has confused the left and the right wrists, which means that step205will determine an incorrect 2D pose in which the right wrist is located in the position of the left wrist.

FIG.8Aalso shows a predicted 2D pose270which has been determined for a time point corresponding to the confidence map CM. The predicted 2D pose270is used in a filtering procedure262to generate a corrected confidence map CM′. In the predicted 2D pose270, the right wrist (open circle) is located on the right side of the object. The filtering procedure262operates a filter function, which is given by the predicted 2D pose270, on CM to generate CM′. In CM′, the confidence values are larger in region271A′ than in region271B′, and step205will correctly determine the location of the right wrist.

FIG.8Bis a flow chart of an example filtering procedure262that may be part of the post-processing step205. Step262A comprises obtaining a set of candidate locations of a predefined feature within an analysis area and confidence values for the candidate locations. The candidate locations and candidate values have been determined by the feature detection step204and is, for example, represented as a confidence map CM. Step262B comprises obtaining one or more predicted 2D poses, which are directly or indirectly given by the PPD. Step262C selectively modifies the set of confidence values for a predefined feature based on the location of a corresponding predefined feature in the predicted 2D pose, and based on a confidence or probability value of the predicted 2D pose, if available. Step262D comprises determining the location of the respective predefined feature based on the modified confidence values.

Step262C may be implemented to determine a filter function based on the predicted 2D poses from step262B and operating the filter function on the confidence values. The filter function may be configured to increase candidate values near the location of the predefined feature in the predicted 2D pose in relation to other candidate values. The filter function may be defined in many different ways and may be algebraic or rule based. In one example, the filter function applies a respective weight to the candidate values, for example by multiplication, where the weight decreases with the distance between the candidate value and the location of the predefined feature in the predicted 2D pose. In a variant, the filter function is determined based on one or more predicted 3D poses, obtained by step262B.

It may be noted that plural confidence maps may be obtained in step262A, that these confidence maps may be selectively modified in step262C based on the predicted pose(s) obtained in step262B, and that the feature location may be determined based on the thus-modified confidence maps in step262D. For example, confidence maps for different predefined features may be processed by steps262C and262D to determine the location of one predefined feature. For example, inFIG.8A, a confidence map for the left wrist and a confidence map for the right wrist may be filtered and analyzed for determining the location of the right wrist.

The embodiments described with reference toFIGS.8A-8Bare applicable to both the partitioned system inFIG.3Aand the centralized system inFIG.3B.

In fact, the above-described validation and/or filtering procedure based on PPD may be used independently of object detection, to improve pose detection in an image.FIG.8Cis a flow chart of an example image processing method200′ in accordance with an embodiment. The method200′ may be performed by any device in the system ofFIG.1A, for example the camera2or the reconstruction device3, if present. Step201comprises receiving or inputting an image I. As indicated by dashed lines, the method200′ may or may not comprise an object detection step202. If present, step202may perform any type of object detection. Step204comprises processing the image I, optionally within an analysis area determined by step202, for detection of features. Step204may be similar to step204in method200. Step205A comprises receiving or inputting PPD, which may be generated as described above for method200or in any other way. Step205B comprises performing an evaluation based on the PPD received in step205A, to generate and output a detected 2D pose (2DP). Step205B may perform the above-described validation and/or the filtering procedure262(FIG.8B).

FIG.9is a non-limiting example of an algorithm that may be implemented by step221to predict 3D poses of an object based on a time sequence of detected 3D poses. In the illustrated example, a branched timeline (“prediction tree”) of predicted 3D poses is constructed. The algorithm inFIG.9may be suitable for implementation in resource constrained systems. The algorithm is exemplified for a human object but is also applicable to animals or inanimate objects. Step301comprises defining a skeleton for a tracked 3D object (T3DO) by specifying: a root joint (RJ), one or more pairs of joints, each making up a unique limb/edge in a tree structure, zero or more sets of symmetrical limbs (SL), a weight distribution (WD) that defines how much of the body weight that is centered around a specific joint, an angular uncertainty range (AUR) that defines how much the angular velocity (AV) for a joint may change at a random point in time, angular limits (AL) that define the range of possible angles for a joint, and optional overflow joints (OJ) for each joint. The OJs are joints over which the remainder of the angles that are outside the AL of a joint will be distributed, for example evenly or by specified weights. Step302comprises obtaining a time sequence of detected 3D poses of a T3DO. This time sequence may be generated by tracking detected 3D poses of an object over time. Step303comprises starting at the RJ and traversing all limbs to calculate the current 3D angles of each joint (3DJA) as the relation between the current limb and the ancestor (previous) limb. For the RJ, the vertical axis of the scene coordinate system (SCS) may be considered the ancestor limb. Step304comprises calculating the motion vector (MV) of the center of gravity (CoG) in the SCS, for example by first calculating the CoG as the mean position of all joints weighted by each joint's WD and then taking an exponential moving average of the difference of the CoG positions. Step305comprises calculating the AV for each joint in the pose, for example using an exponential moving average of the difference in joint angles between temporally adjacent poses. Step306comprises calculating the mean limb length (mLL) for each limb by averaging the lengths in all seen poses for the T3DO, taking symmetry into account by treating SL as the same limb. Then, steps307-316are performed repeatedly in discrete time steps. Step307comprises calculating new 3D joint angles from the previous angles plus the AV for the respective joint. If it is detected, by step308, that a new 3DJA exceeds its AL, the timeline for the T3DO is branched by steps309-310. In the original branch, the affected joint(s) may stop moving, and in the new branch the angle values, which could not be applied to the affected joint(s) as a result of the AL, may be distributed on the OJs of the affected joint(s). Thus, the AVs that exceed the AL are set to zero in the original branch and are unchanged in the new branch. The remainder outside the AL may be distributed to the OJs in a recursive fashion, for example so that AL is always respected, and remainders are distributed to the corresponding OJ when given.

To further explain steps308-310in a simplified example, consider a human body with a straight arm moving upwards so that only the shoulder joint has non-zero AV. When the angle of the shoulder joint hits its AL, the AV no longer affects the shoulder joint angle but instead overflows to the OJ of the shoulder joint, for example the elbow joint, making the arm bend. The timeline is then split into an original branch in which the arm is still with the shoulder angle at the AL, and a new branch in which the arm keeps on bending at the elbow joint.

It may be noted that steps308-310may consider several T3DOs in the calculations, for example to account for collisions or identify collaborative actions such as a handshake, a chase, etc.

In some embodiments, a new branch may also be created at random. For example, the AV for a randomly chosen joint may be multiplied by a random value in the joint's AUR.

Step311comprises determining a new 3D pose (new3DP) using the MLLs and the new3DJAs. Step312comprises calculating a center location (CL) for CoG alignment by extrapolation from the previous CoG, for example by adding MV to the previous CoG. Step313comprises calculating CoG for the new3DP, in correspondence with step304. Step314comprises aligning the new3DP with CL by using the new CoG as anchor. Step315appends the new3DP to the timeline, and step316proceeds to the next time step and returns to step307. When steps307-316have been repeated until an end time (cf. PTW inFIG.7B), step317selects a next time step of one of the branches and returns to step307, which proceeds to predict 3D poses for the branch. When steps307-316have been repeated for all branches until the end time, the method221may performed for another T3DO until predictions are generated for all T3DOs.

The structures and methods disclosed herein may be implemented by hardware or a combination of software and hardware. In some embodiments, such hardware comprises one or more software-controlled computer resources.FIG.10schematically depicts such a computer resource, which may represent the camera2or the reconstruction device3. The computer resource comprises a processing system101, computer memory102, and a communication interface103for input and/or output of data. Depending on implementation, the computer resource may also include an image sensor51, as indicated by dashed lines. The communication interface103may be configured for wired and/or wireless communication. The processing system101may, for example, include one or more of a CPU (“Central Processing Unit”), a DSP (“Digital Signal Processor”), a microprocessor, a microcontroller, an ASIC (“Application-Specific Integrated Circuit”), a combination of discrete analog and/or digital components, or some other programmable logical device, such as an FPGA (“Field Programmable Gate Array”). A control program102A comprising computer instructions is stored in the memory102and executed by the processing system101to perform any of the methods, procedures, operations, functions or steps described in the foregoing. As indicated inFIG.10, the memory102may also store control data102B for use by the processing system102. The control program102A may be supplied to the computing resource on a computer-readable medium110, which may be a tangible (non-transitory) product (for example, magnetic medium, optical disk, read-only memory, flash memory, etc.) or a propagating signal.

Although the objects are represented as human individuals in the foregoing examples, the disclosed technique is applicable to any type of object, be it living or inanimate.

The techniques disclosed and exemplified herein have a variety of applications such as 3D vision inspection, product assembly, goods inspection, human-computer interaction, video surveillance, sports broadcasting, industry robot control, navigation, etc. The present Applicant also contemplates to arrange a monitoring system that implements the disclosed technique to track individuals in an exercise situation, for example in a gym. For example, the monitoring system may track how the individuals move around the gym, identify activity and count repetitions by use of 3D poses of the individuals, for example representing joints of the individuals, and store corresponding exercise data for the respective individual in a database for access by the respective individual or another party.

In the following, clauses are recited to summarize some aspects and embodiments of the invention as disclosed in the foregoing.

Clause 1. A processing device configured to: obtain a sequence of images of a scene captured by an image sensor (51); determine an analysis area ([BB]) for an object in a respective image in the sequence of images; and process the respective image within the analysis area ([BB]) for detection of predefined features of the object, wherein the processing device is further configured to receive pose prediction data (PPD) which represents predicted poses of the object as a function of time, and wherein the processing device is configured to determine the analysis area ([BB]) based on the pose prediction data (PPD).

Clause 2. The processing device of clause 1, which is further configured to determine, based on the pose prediction data (PPD), a predicted pose of the object at a time point associated with the respective image, wherein the processing device is configured to determine the analysis area ([BB]) based on the predicted pose.

Clause 3. The processing device of clause 2, wherein the analysis area ([BB]) is defined as a bounding box that surrounds the predicted pose when the predicted pose is spatially matched to the image.

Clause 4. The processing device of clause 2 or 3, which is configured to determine the predicted pose of the object at a time point associated with the respective image by temporally interpolating two or more predicted poses in the pose prediction data (PPD).

Clause 5. The processing device of any preceding clause, wherein the predicted poses in the pose prediction data (PPD) are defined in a 3D coordinate system (30) associated with the scene, wherein the processing device is further configured to: spatially match the predicted poses to the image by projecting the predicted poses into a 2D coordinate system (32) associated with the respective image.

Clause 6. The processing device of any one of clauses 1-4, wherein the predicted poses in the pose prediction data (PPD) are defined in a 2D coordinate system (32) associated with the respective image.

Clause 7. The processing device of any preceding clause, which is further configured to: process locations of the predefined features that are detected within the analysis area ([BB]) for the respective image to determine a detected 2D pose of the object.

Clause 8. The processing device of clause 7, wherein the pose prediction data (PPD) defines a branching time point (BTP), which separates a progressing time sequence of predicted poses into two or more alternative sequences of poses, wherein the processing device is further configured to: perform an evaluation of the detected 2D pose of the object in an image associated with a time point at or near the branching time point (BTP) in relation to predicted poses in the two or more alternative sequences, select one of the alternative sequences based on the evaluation, and determine the predicted pose of the object based on said one of the alternative sequences.

Clause 9. The processing device of any preceding clause, which is further configured to: evaluate the predefined features that are detected within the analysis area ([BB]) for the respective image in relation to at least one predicted pose in the pose prediction data (PPD).

Clause 10. The processing device of any preceding clause, which is configured to determine, by processing the respective image within the analysis area ([BB]), a set of candidate locations of the predefined features in the respective image and confidence values for the candidate locations, and wherein the processing device is configured to: selectively modify the confidence values based on locations of corresponding predefined features in the at least one predicted pose in the pose prediction data (PPD), and determine a set of detected locations of the predefined features in the respective image based on the selectively modified confidence values.

Clause 11. The processing device of clause 10, wherein the set of candidate locations and the confidence values are given as confidence maps that indicate a confidence value of a respective predefined feature being located at a respective candidate location in the image.

Clause 12. The processing device of any preceding clause, further comprising the image sensor (51).

Clause 13. The processing device of any preceding clause, which is further configured to determine a sequence of 3D poses ([3D]) of the object based on images of the scene captured by a plurality of image sensors, and generate the pose prediction data based on the sequence of 3D poses ([3D]).

Clause 14. A system comprising a plurality of processing devices according to any preceding clause, wherein the processing devices are configured to obtain a respective sequence of images of the scene from a respective image sensor (51), the system further comprising a prediction module (63), which is configured to generate the pose prediction data (PPD) and provide the pose prediction data (PPD) to the plurality of processing devices.

Clause 15. The system of clause 14, wherein the prediction module (63) is configured to operate a pose prediction algorithm on a sequence of 3D poses ([3DP]) of the object to determine predicted 3D poses ([3DP]*) of the object as a function of time within a prediction time window (PTW), and to generate the pose prediction data (PPD) to represent at least a subset of the predicted 3D poses ([3DP]*).

Clause 16. The system of clause 15, further comprising a triangulation module (61) which is configured to obtain sequences of detected 2D poses from the plurality of processing devices, and to calculate the sequence of 3D poses ([3DP]) of the object by operating a triangulation algorithm on time-synchronized 2D poses among the sequences of detected 2D poses.

Clause 17. The system of clause 16, wherein the triangulation module (61) and the prediction module (63) are physically separated from the processing device.

Clause 18. The system of clause 16 or 17, wherein each of the processing devices is configured to transfer a sequence of detected 2D poses corresponding to the sequence of images to the triangulation module (61).

Clause 19. The system of clause 18, further comprising an estimation module (64), which is configured to estimate a latency period (LT) from a transfer of the sequence of detected 2D poses from one of the processing devices to a receipt of the pose prediction data (PPD) by said one of the processing devices, wherein the prediction module (63) is configured to generate the pose prediction data (PPD) based on the latency period (LT).

Clause 20. The system of clause 19, wherein the prediction module (63) is configured to set the prediction time window (PTW) based on the latency period (LT).

Clause 21. The system of any one of clauses 16-20, wherein at least a subset of the processing devices is further configured to, intermittently, process one or more images in the sequence of images for detection of new objects and determine one or more 2D poses for a respective new object that is detected in the one or more images, and wherein said at least a subset of the processing devices is configured to transmit the one or more 2D poses for the respective object to the triangulation module (61).

Clause 22. The system of any one of clauses 15-21, which is configured to store, output or process the sequence of 3D poses ([3DP]).

Clause 23. A computer-implemented method for image processing, said method comprising: obtaining (201) a sequence of images of a scene captured by an image sensor (51); determining (202) an analysis area for an object in a respective image in the sequence of images; and processing (204) the respective image within the analysis area for detection of predefined features of the object; wherein the method further comprises receiving (202A) pose prediction data (PPD) which represents predicted poses of the object as a function of time, wherein the analysis area is determined (202B) based on the pose prediction data (PPD).

Clause 24. A processing device configured to: obtain a sequence of images of a scene captured by an image sensor (51); process the respective image for detection of predefined features of an object; receive pose prediction data (PPD) which represents predicted poses of the object as a function of time; and evaluate the predefined features that are detected for the respective image in relation to at least one predicted pose in the pose prediction data (PPD) to determine a pose of the object.

Clause 25. A computer-implemented method for image processing: comprising: obtaining (201) a sequence of images of a scene captured by an image sensor (51); processing (204) the respective image for detection of predefined features of an object; receiving (205A) pose prediction data (PPD) which represents predicted poses of the object as a function of time; and evaluating (205B) the predefined features that are detected for the respective image in relation to at least one predicted pose in the pose prediction data (PPD), to determine a pose of the object.

Clause 26. A computer-readable medium comprising computer instructions (102A) which, when executed by a processor (101), cause the processor (101) to the perform the method of clause 23 or 25.