SYSTEMS AND METHODS FOR TEMPORAL AUTOCALIBRATION IN A CAMERA SYSTEM

Systems and methods for calibrating an imaging device. A processing circuit identifies first timestamps of one or more events in one or more first image frames of a first imaging device, and identifies one or more corresponding events in one or more second image frames. Second timestamps may be determined for the corresponding events. A calibration value may be calculated based on a difference between the first and second timestamps. In one embodiment, the processing circuit also identifies a first plurality of events in a first sequence of image frames transmitted by a first imaging device, and selects a second plurality of events in a second sequence of image frames by a second imaging device that minimizes discrepancy between the first plurality of events and the second plurality of events. A calibration value is computed based on the timestamps of the matched events.

FIELD

Aspects of embodiments of the present disclosure relate to a camera system, and in particular, to automatic temporal calibration of multiple cameras in a multi-camera system.

BACKGROUND

In many different applications the correct work of multicamera setup requires accurate temporal calibration. Such setups can lose the calibration after, for example, restart of any one of the cameras and/or due to differences in the running frequency of the internal clocks of the cameras. Thus, calibration metrics can drift over time. Manually recalibrating the cameras can be a time-consuming process. It is therefore desirable to have a system and method that automatically recalibrates the cameras from time to time.

SUMMARY

An embodiment of the present disclosure is directed to a method for calibrating an imaging device. A processing circuit processes a first image frame captured by a first imaging device, identifies an event in the first image frame, and determines a first time value of the event. The processing circuit also processes a second image frame captured by a second imaging device, identifies the event in the second image frame, and determines a second time value of the event. The processing circuit calculates a calibration value based on a difference between the first time value and the second time value, and adjusts a time value of a third image frame provided by the second imaging device based on the calibration value.

According to one embodiment, the event is a start or stop of movement of an object through a scene.

According to one embodiment, the first time value is a first timestamp of the first image frame, and the second time value is a second timestamp of the second image frame.

An embodiment of the present disclosure is also directed to a method for calibrating an imaging device that includes: processing, by a processing circuit, a first sequence of image frames transmitted by a first imaging device; identifying, by the processing circuit, a first plurality of events in the first sequence; processing, by the processing circuit, a second sequence of image frames transmitted by a second imaging device; selecting, by the processing circuit, a second plurality of events in the second sequence that minimizes discrepancy between the first plurality of events and the second plurality of events; determining, by the processing circuit, a first plurality of time values corresponding to the first plurality of events, and a second plurality of time values corresponding to the second plurality of events; computing, by the processing circuit, a calibration value based on a difference between the first plurality of time values and the second plurality of time values; and adjusting, by the processing circuit, a time value of a third image frame provided by the second imaging device based on the calibration value.

According to one embodiment, the first plurality of events include start and stop movements of an object through a scene.

According to one embodiment, the first plurality of time values include timestamps of the first sequence of image frames, and the second plurality of time values include timestamps of the second sequence of image frames.

According to one embodiment, the calibration value is an average of a first difference and a second difference, wherein the first difference is between times of first events selected from the first plurality of events and the second plurality of events, and the second difference is between times of second events selected from the first plurality of events and the second plurality of events.

According to one embodiment, the selecting of the second plurality of events includes iteratively aligning the first plurality of events to different ones of the second plurality of events, and iteratively computing discrepancy measurements based on the aligning.

An embodiment of the present disclosure is also directed to a method for calibrating an imaging device comprising: processing, by a processing circuit, a first sequence of image frames transmitted by a first imaging device, wherein one or more of the first sequence of image frames includes first motion-blurred images; computing, by the processing circuit, a first blur kernel for a first image frame in the first sequence of image frames; computing, by the processing circuit, a first movement profile for the first blur kernel; processing, by a processing circuit, a second sequence of image frames transmitted by a second imaging device, wherein one or more of the second sequence of image frames includes second motion-blurred images; computing, by the processing circuit, a second blur kernel for a second image frame in the second sequence of image frames; computing, by the processing circuit, a second movement profile for the second blur kernel; computing, by the processing circuit, a calibration value that minimizes discrepancy between the first movement profile and the second movement profile; and adjusting, by the processing circuit, a timestamp of a third image frame provided by the second imaging device based on the calibration value.

According to one embodiment, the first movement profile and the second movement profile encode linear movements.

An embodiment of the present disclosure is further directed to an imaging system comprising a first imaging device, a second imaging device, and a processing system coupled to the first imaging device and the second imaging device. The processing system comprises a processor and memory storing instructions that, when executed by the processor, cause the processor to perform: processing a first image frame captured by a first imaging device; identifying an event in the first image frame; determining a first time value of the event; processing a second image frame captured by a second imaging device; identifying the event in the second image frame; determining a second time value of the event; calculating a calibration value based on a difference between the first time value and the second time value; and adjusting a time value of a third image frame provided by the second imaging device based on the calibration value.

An embodiment of the present disclosure is also directed to an imaging system comprising a first imaging device, a second imaging device, and a processing system coupled to the first imaging device and the second imaging device. The processing system comprises a processor and memory storing instructions that, when executed by the processor, cause the processor to perform: processing a first sequence of image frames transmitted by a first imaging device; identifying a first plurality of events in the first sequence; processing a second sequence of image frames transmitted by a second imaging device; selecting a second plurality of events in the second sequence that minimizes discrepancy between the first plurality of events and the second plurality of events; determining a first plurality of time values corresponding to the first plurality of events, and a second plurality of time values corresponding to the second plurality of events; computing a calibration value based on a difference between the first plurality of time values and the second plurality of time values; and adjusting a time value of a third image frame provided by the second imaging device based on the calibration value.

An embodiment of the present disclosure is further directed to an imaging system comprising a first imaging device, a second imaging device, and a processing system coupled to the first imaging device and the second imaging device. The processing system includes a processor and memory storing instructions that, when executed by the processor, cause the processor to perform: processing a first sequence of image frames transmitted by a first imaging device, wherein one or more of the first sequence of image frames includes first motion-blurred images; computing a first blur kernel for a first image frame in the first sequence of image frames; computing a first movement profile for the first blur kernel; processing a second sequence of image frames transmitted by a second imaging device, wherein one or more of the second sequence of image frames includes second motion-blurred images; computing a second blur kernel for a second image frame in the second sequence of image frames; computing a second movement profile for the second blur kernel; computing a calibration value that minimizes discrepancy between the first movement profile and the second movement profile; and adjusting a timestamp of a third image frame provided by the second imaging device based on the calibration value.

It should be appreciated that the claimed system and method enable temporal autocalibration of a camera in a multi-camera setup based on one or more events captured in a scene. It should further be appreciated that the claimed autocalibration system and method do not require special calibration objects and avoid disabling of the multicamera setup to do the calibration.

These and other features, aspects and advantages of the embodiments of the present disclosure will be more fully understood when considered with respect to the following detailed description, appended claims, and accompanying drawings. Of course, the actual scope of the invention is defined by the appended claims.

DETAILED DESCRIPTION

In a multi-camera system, an object moving through a scene may be captured by multiple cameras from multiple viewpoints at a known framerate. In order to restore certain information about the moving object, including, for example, position of the object in six degrees of freedom (6DOF), direction and speed of the object, and the like, it is desirable for the cameras to be temporally and spatially calibrated. Temporal calibration may be synchronization of timestamps given to captured video image frames by a group of N cameras such that the multi-camera system imaging the same scene can identify N different video image frames depicting the object at a same moment in time from different viewpoints.

Initial calibration of the cameras may eventually drift over time due, for example, the cameras being turned off and on and/or due to differences in the running frequencies of the internal clocks (e.g., due differences in frequencies within engineering tolerances, due to temperature differences, and due to aging of components of the clock circuits), and the initial calibration metric may lose its relevance. Recalibrating the cameras manually may be a time-consuming process. Accordingly, it is desirable to have an automatic system and method for recalibrating the cameras.

In general terms, embodiments of the present disclosure are directed to a multi-camera system that is configured to engage in temporal autocalibration and/or calibration refinement (collectively referred to as autocalibration) of N cameras of the system, based on one or more detected events of a captured scene. In one embodiment, movement of an object across the scene may be used for the temporal autocalibration. For example, start and/or stop movements of an object transported on a conveyor belt may be monitored and used for the temporal autocalibration. Although the various embodiments described herein assume that the monitored movement is that of an object on a conveyor belt, a person of skill in the art should recognize that the monitored movement may be any linear movement of the object across a scene.

In one embodiment, two or more cameras in the multi-camera system capture images of the object moving across the scene, from multiple viewpoints. In one embodiment, an assumption is made that the image frames are captured with low enough (or short enough) exposure time so that any motion blur may be neglected in the captured video frames. Based on this assumption, temporal autocalibration may be performed, according to one embodiment, based on one or more detectable events captured by the video frames. The detectable event may be a detected start and/or stop of movement of the object on a conveyor belt.

In one embodiment, a first autocalibration methodology is employed by the camera system to calculate a temporal calibration value for the cameras when only a single event is detected. In this regard, one of the N cameras is selected as a main camera to which other cameras are synchronized. In synchronizing a support camera of the N cameras with the main camera, timestamps of the video frames that captured the detectable event are identified for both the main camera and the support camera. A difference of the two timestamps is set to be as the calibration value for the support camera. According to the first methodology, the temporal calibration value is equal or greater than the time per frame.

In one embodiment, a second autocalibration methodology is employed by the camera system to calculate a temporal calibration value for the cameras when multiple events are detected. The second autocalibration methodology may allow detection of temporal discrepancies between cameras that may be below the time per frame (e.g., temporal discrepancies that are shorter than the length of a frame). Such a discrepancy may be referred to as a subframe discrepancy, and the calculated temporal calibration value may be referred to as a subframe temporal calibration value.

In one embodiment, the calculating of the subframe temporal calibration value includes computing a temporal calibration value that best aligns the events captured by the main camera with the events captured by the support camera. In one embodiment, one or more tentative subframe temporal calibration values are computed based on one or more attempts to match the events detected by the two cameras, and an error/discrepancy metric is computed based on each of the computed tentative subframe temporal calibration values. The error metric may be the sum of discrepancy values of the matched events as corrected by the tentative subframe temporal calibration value. In one embodiment, the computed subframe temporal calibration value that minimizes the error metric is selected as the actual subframe temporal calibration value for the support camera.

In one embodiment, a third autocalibration methodology is used by the camera system for calculating a temporal calibration value in the presence of motion blur effect on the video frames. The motion blur may be due to the object moving with respect to the camera during the recording of a single exposure. In this case, the object moving with respect to the camera will look blurred or smeared along the direction of relative motion. In one example, the longer the exposure window/time, the bigger the motion blur.

In one embodiment, the camera system computes an estimated motion blur kernel that describes the object motion during an exposure window. An unblurred or latent image may be estimated via a deconvolution algorithm using the estimated motion blur kernel and the captured blurry image.

In the event of linear movement through the scene, the motion blur kernel may take the form of one or more linear strokes. The linear strokes may be converted into a one-dimensional (1D) movement profile. In computing the temporal calibration value, an aggregated movement profile of the main camera for multiple frames is compared against the aggregated movement profile of the support camera for multiple frames, and a calibration value is selected that minimizes the discrepancy/error between the two aggregated movement profiles.

In one embodiment, one or more of the autocalibration methodologies are run concurrently with each other, or in series. For example, the first autocalibration methodology may be run to roughly compute a first temporal calibration value, and the second autocalibration may also be run to compute any subframe discrepancy that is less than the time per frame.

FIG.1is a schematic block diagram of an imaging system configured to execute one or more autocalibration methodologies according to one embodiment. The system may include a first/main camera (first imaging device)10and one or more second/support cameras30a-30c(second imaging devices) (collectively referenced as30).

In one embodiment, the main camera10includes a stereo camera. Examples of stereo cameras include camera systems that have at least two monocular cameras spaced apart from each other along a baseline, where the monocular cameras have overlapping fields of view and optical axes that are substantially parallel to one another. The support cameras30may be stereo cameras, monocular cameras, or combinations thereof (e.g., some stereo support cameras and some monocular support cameras). The main camera10and the support cameras30may use the same imaging modalities or different imaging modalities. Examples of imaging modalities include monochrome, color, infrared, ultraviolet, thermal, polarization, and combinations thereof.

In the embodiment ofFIG.1, the main camera10and the support cameras are arranged such that their field of view12captures a scene with an object22transported on a carrying medium24such as, for example, a conveyor belt. Although a conveyor belt is used as an example, embodiments of the present disclosure are not limited thereto, and may include other carrying mediums such as, for example, a movable arm of a robot and/or the like.

In one embodiment, the main camera10is located above the carrying medium24(e.g., spaced apart from the objects22along the direction of gravity). In some embodiments, the main camera is located at other positions and/or orientations. For example, the main camera may be arranged to have a downward angled view of the objects22.

In some embodiments, the support cameras30are arranged at different poses around the carrying medium24transporting the objects22. Accordingly, each of the support cameras30, e.g., first support camera30a, second support camera30b, and third support camera30c, captures a different view of the objects22from a different view point (e.g., a first viewpoint, a second viewpoint, and a third viewpoint, respectively). WhileFIG.1shows three support cameras30, embodiments of the present disclosure are not limited thereto, and may include, for example, at least one support camera30and may include more than three support cameras30.

In one embodiment, the main camera10and the support cameras30each include a clock used to assign timestamps to the image frames captured by the corresponding camera. Although initially the clocks of the various cameras may be calibrated/synchronized to each other, over time, the clocks may become uncalibrated/unsynchronized. For example, one of the cameras may be turned off and then on, causing that camera to become temporally uncalibrated. As another example, one camera may be regularly exposed to direct sunlight while another camera remains shaded, such that they consistently operate at different temperatures or component aging at different rates, thereby causing clock drift due to shifts in the resonant frequencies of their clock circuits.

In one embodiment, the multi-camera system includes an autocalibration system100configured to perform temporal autocalibration of one or more of the cameras. The autocalibration may occur on a scheduled basis (e.g. every hour, day, week, etc.), or upon detecting a certain event. For example, the event may be one of the cameras10,30turning on after being off.

In one embodiment, the autocalibration system100may be configured to compute/estimate temporal autocalibration values for the main and/or support cameras10,30, based on information captured by the cameras. The autocalibration system100may include processing circuits or electronic circuits configured to compute or estimate temporal autocalibration values for the main and/or support cameras10,30. Types of electronic circuits may include a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence (AI) accelerator (e.g., a vector processor, which may include vector arithmetic logic units configured efficiently perform operations common to neural networks, such dot products and softmax), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), or the like. For example, in some circumstances, aspects of embodiments of the present disclosure are implemented in program instructions that are stored in a non-volatile computer readable memory where, when executed by the electronic circuit (e.g., a CPU, a GPU, an AI accelerator, or combinations thereof), perform the operations described herein to compute the temporal autocalibration values.

The operations performed by the autocalibration system100may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, or the like) or may be allocated between multiple electronic circuits (e.g., multiple GPUs or a CPU in conjunction with a GPU). The multiple electronic circuits may be local to one another (e.g., located on a same die, located within a same package, located in one or more of the main camera10or support cameras30, or located within a same embedded device or computer system) and/or may be remote from one other (e.g., in communication over a network such as a local personal area network such as Bluetooth®, over a local area network such as a local wired and/or wireless network, and/or over wide area network such as the internet, such a case where some operations are performed locally and other operations are performed on a server hosted by a cloud computing service). One or more electronic circuits operating to implement the autocalibration system100may be referred to herein as a computer or a computer system, which may include memory storing instructions that, when executed by the one or more electronic circuits, implement the systems and methods described herein.

In one embodiment, the autocalibration system100is configured to control the main camera10and the support cameras30to stream images of the objects transported via the carrying medium24from different viewpoints. The autocalibration system100may identify one or more events from the streamed images, and the identified events may then be used to calibrate the cameras10,30so that they are temporally synchronized. In this regard, one of the cameras10,30may be selected as a main camera to serve as the basis of the temporal calibration, and the remaining cameras may be calibrated to the main camera. The selection of main camera may be, for example, random. For ease of description, the main camera10is assumed to be the main camera, and the support cameras30are assumed to be the cameras that are calibrated to the main camera.

In one embodiment, the autocalibration system100processes the videos streamed by the main camera10, support cameras30, or a combination of the main camera and the support cameras, during a span of time, for identifying an event. The monitored span of time may be, for example, the time span between a switch on and switch off events between two cameras. The monitored event may be, for example, starting or stopping of movement of one of the objects22on the carrying medium24. Other events that are simultaneously detectable by the main camera and the support cameras may also be monitored and identified in addition or in lieu of start and stop events. For example, brief flashes of light (that may be intentionally projected to help with synchronization), jostling in a direction perpendicular to the movement of the carrying medium24, reversal of direction of movement of the carrying medium24, and/or the like may also be monitored and used for calibrating the cameras.

In one embodiment, the autocalibration system100is configured with an optical flow algorithm, key point detection algorithm, and/or the like, for identifying a particular event. For example, the optical flow algorithm may be invoked for detecting the movement of an object between successive image frames of a video. The direction of movement and velocity may produce optical flow. Thus, through the detection of optical flow, the movement of an object may be determined for identifying moments in which the movements start, end, and/or take another predefined action.

In general terms, optical flow relates to the distribution of apparent velocities of movement of brightness patterns in an image (see, e.g., Horn, Berthold K P, and Brian G. Schunck. “Determining optical flow.”Artificial intelligence17.1-3 (1981): 185-203.). One common use of optical flow relates to detecting the movement of objects between successive image frames of a video, such as detecting the motion of a soccer ball based on the change of position of the brightness patterns associated with the ball (e.g., black and white patches) from one frame to the next. An optical flow map may represent the velocities of each pixel value in a first image frame to a corresponding pixel in the second image frame. For example, the brightness at a point (x, y) in the first image at time t may be denoted as E(x, y, t), and this pixel may move by some distance (Δx, Δy) from time t associated with the first image frame to time t+Δt associated with the second frame. Accordingly, the optical flow map may include a velocity (u, v) for each point (x, y) in the first image frame, where u=dx/dt and v=dy/dt. One aspect of algorithms for computing optical flow fields relates to determining correct correspondences between pairs of pixels in the two images. For example, for any given point (x, y) in the first image, there may be many pixels in the second image having the same brightness, and therefore an optical flow algorithm will need to determine which pixel in the second image corresponds to the point (x, y) of the first image, even if the corresponding point in the second image has a different brightness or appearance due to changes in lighting, noise, or the like.

Movement of an object may also be detected using a keypoint detection algorithm. The keypoint detection algorithm may be invoked to detect keypoints (e.g. unique parts) of the object22. The keypoints may be detected using, for example, a classical keypoint detector (e.g., scale-invariant feature transform (SIFT), speeded up robust features (SURF), gradient location and orientation histogram (GLOH), histogram of oriented gradients (HOG), basis coefficients, or Haar wavelet coefficients) or a trained deep learning keypoint detector such as a trained convolutional neural network using HRNet (Wang, Jingdong, et al. “Deep high-resolution representation learning for visual recognition.”IEEE transactions on pattern analysis and machine intelligence(2020).) with a differential spatial to numerical (DSNT) layer and Blind Perspective-n-Point (Campbell, Dylan, Liu, and Stephen Gould. “Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization.”European Conference on Computer Vision. Springer, Cham, 2020).

In one embodiment, the autocalibration system100monitors the position of the keypoints in successive frames of the video streamed by the main camera10and/or support camera30for determining, for example, start or stop of movement of the object. For example, if the position of the keypoints identified for the object22remain the same in two consecutive frames, but change in a following frame, a “start movement” event may be identified for the following frame.

In one embodiment, the autocalibration system100employs one or more autocalibration methodologies for calibrating the support cameras30to the main camera10. The methodology that is selected may be based on an identified number of detectable events in the streamed video during a span of time, and further based on the identified camera exposure time. In one embodiment, the autocalibration system100employs rough temporal calibration if a single event is detected in the streamed video(s) during the span of time. In this regard, in response to detecting the single event by the autocalibration system100(e.g. in the main camera10), a matching event is identified in an image frame streamed by one of the other cameras (e.g. one of the support cameras30). A timestamp of the event/moment m when the event is identified on the video stream of camera i may be defined as tm,i(e.g., the timestamp of the frame captured by camera i corresponding to the moment m of the event). The rough calibration value for camera i (Δti,Rough) may then be computed by subtracting the timestamp of the moment of support camera i, from the timestamp of the moment of the main camera 0 (e.g., the timestamp of the frame captured by main camera 0 corresponding the moment m of the event), as follows:

In one embodiment, if multiple events are detected by the autocalibration system100during the span of time, the autocalibration system100engages in subframe temporal calibration for each support camera30in relation to the main camera10. In this regard, in calculating a subframe temporal calibration value for each support camera i30, the autocalibration system100processes a first sequence of image frames transmitted by the main camera10, and a second sequence of image frames transmitted by support camera i30. The autocalibration system100may detect multiple events in the first sequence of image frames (e.g. one or more start and stop events). In one embodiment, the autocalibration system100takes a first plurality of events detected by the main camera10and matches it with a second plurality of events detected by support camera i30. For example, the autocalibration system may take the first two detected events of the main camera10, and match it to the last two corresponding events (e.g. by matching a start event with a start event, and stop event with a stop event) of the support camera i30. In one embodiment, the autocalibration system100attempts different matches until it finds a match that minimizes a discrepancy computation (or error) between the timestamps of the first plurality of events and the second plurality of events. The matching may be done, for example, by trying all possible combinations until an optimal match is found.

In one embodiment, the autocalibration system100computes a tentative calibration value using a first tentative match of the events. The tentative calibration value for camera i(Δti) may be an average of the temporal discrepancies between the timestamps of the first and second plurality of events, according to the following equation:

where n is the number of matched events, te,0is the timestamp of an event detected by the main camera10, and te,iis the timestamp of a matched event detected by camera i.

Based on the tentative calibration value, the autocalibration system100computes a discrepancy/error E value based on the tentative calibration value according to the following formula:

In one embodiment, different matches of the events detected by the main camera10are attempted, and a discrepancy value computed for each of the attempts. For example, during a second iteration of the match, the matching process may move to match different (e.g., earlier) set of events detected by camera i with the events captured by the main camera10, and attempt more matches of events between the main camera10and camera i. A set of matches that provides a minimal discrepancy value is selected, and the associated calibration value is set to be the final calibration value for camera i.

In one embodiment, the autocalibration system100employs a motion blur-based temporal calibration in the event that the image frames transmitted by the cameras include motion-blurred images. The motion blur may be due to motion of an object being recorded during a single exposure. In one embodiment, the exposure time is assumed to be more than 50% of the time per frame in order to guarantee overlap of exposures between the main camera and the support camera. That is, if the exposure time is equal or less than 50%, the problem may become ill-posed. For example, if camera 1 is exposed to the first half of the time per frame, and camera 2 is exposed to the second half, then there is no overlap between the two cameras. Thus, in one embodiment, the motion blur-based temporal calibration is employed when the exposure time is more than 50% in order to guarantee at least some overlap between the cameras.

In one embodiment, the autocalibration system100processes the blurred image frames transmitted by the main10and the support cameras30, and computes a motion blur kernel for each image frame based on the motion-blurred image captured in the frame. The motion blur kernel may encode the movement of the camera and objects22in the scene. In the embodiment where the movement of the objects22is linear (e.g. on the carrying medium24) through the scene, the blur kernel may capture the movement as one or more linear strokes.

In one embodiment, the autocalibration system100converts the blur kernel into a movement profile. In one embodiment, the movement profile xiis a 1 D movement profile defined as a function:

In this regard, the linear stroke(s) in the blur kernel are converted into a vector of value(s) based on the value(s) of the pixel(s) in the motion blur kernel. For example, pixels with higher brightness may indicate a faster movement than pixels with lower brightness, and may be converted to a value that is greater than the pixels with the lower brightness.

In one embodiment, the movement profiles of the various frames of camera i are combined to generate a single combined movement profile. Similarly, the movement profiles of the various frames of the main camera10are combined to generate a single combined movement profile. In one embodiment, a calibration value for camera i(Δti) is computed so that the following error value E is minimized:

where x0and x1are movement profiles for respectively the main camera10and camera i combined from multiple frames, c0and ciare functions equal to 1 when the camera sensor is exposed and, 0 when the sensor is not exposed.

FIG.2is a conceptual diagram of an example stop event200athat may be detected by the main camera10and one of the support cameras30, and used for autocalibration of the one of the support cameras30using the rough autocalibration mechanism according to one embodiment. In the example ofFIG.2, the main camera10transmits a sequence of image frames202acapturing images of an object22in the scene from a first viewpoint. The support camera30also transmits a sequence of image frames202bcapturing images of the object from a second viewpoint. Each image frame202has an exposure time204a,204bwhich, in the example ofFIG.2, is small (e.g. less than 50% of the frame time), so that no motion blur is assumed.

In one embodiment, each image frame202is associated with a timestamp (e.g. T0, T1, etc.). In the example ofFIG.2, the support camera30is offset from the main camera10by a single frame period Tf206. In one embodiment, the autocalibration system100uses the stop event200athat is captured by both cameras for calculating the autocalibration value for the support camera30. In this regard, the main camera10detects the stop event as occurring at timestamp T1. The support camera30, on the other hand, detects the stop event as occurring at timestamp T0. The autocalibration system100takes the timestamps T1and T0from the main camera and the support camera, and computes the autocalibration value as a difference of the two timestamps.

FIG.3is flow diagram of a process for rough temporal autocalibration of a second imaging device (e.g. support camera30) to a first imaging device (e.g. main camera10) according to one embodiment. The process starts, and at block300, the autocalibration system100processes a first image frame transmitted by the first imaging device.

At block302, the autocalibration system100identifies an event in the first image frame. The event may be, for example, start of motion of an object (e.g. object22) on the carrier medium24, stop of motion of the object, or the like. The event may be identified using any algorithm conventional in the art such as, for example, an optical flow algorithm that detects movement (or lack thereof) via change of position of brightness patterns associated with the object from one frame to the next. Another algorithm that may be employed for detecting the event may be a keypoint detection algorithm that may monitor the change of position of the keypoints (e.g. unique features) of the object from one frame to the next.

At block304, the autocalibration system100identifies a first time of the event. The first time may be, for example, the timestamp of the first image frame that is given by the clock of the first imaging device.

The autocalibration system100attempts to identify the same event in an image frame captured by a second imaging device (e.g. support camera30). In this regard, at block306, the autocalibration system100processes a second image frame captured by a second imaging device (e.g. support camera30), and at block308, identifies the event that was detected by the first imaging device, in the second image frame.

At block310, the autocalibration system100identifies a second time of the event. The second time may be, for example, the timestamp of the second image frame that is given by the clock of the second imaging device.

At block312, the autocalibration system100calculates a calibration value based on a difference between the first time and the second time.

At block314, the autocalibration system100adjusts a timestamp of a third image frame provided by the second imaging device based on the calibration value. In this manner, the image frames provided by the second imaging device may be synchronized to the image frames of the first imaging device.

FIG.4is a conceptual diagram of an example of subframe time discrepancy between the main camera10and the support camera30. The support camera30may be described as having a subframe time discrepancy400if the discrepancy is less than the time of a single image frame. In the example ofFIG.3, the time discrepancy between the main camera10and the support camera30is ⅓ of a single frame period. Thus, the main camera10may detect a stop event200bas occurring at timestamp T1. Because of the subframe time discrepancy, the support camera30also detects the stop event200bas occurring at timestamp T1. Thus, the rough autocalibration mechanism may not be sufficient for adjusting for such a subframe time discrepancy in the presence of a single calibration event. The subframe autocalibration mechanism, however, may be used for adjusting for subframe time discrepancies when multiple calibration events are detected.

FIGS.5A-5Bare conceptual diagrams of a process for attempting different tentative alignment of events detected by the main camera10and one of the support cameras30, for performing subframe autocalibration based on the optimal alignment according to one embodiment. In one example, the main camera10provides a sequence of image frames that capture calibration events such as start events400a-400d(collectively referenced as400) and stop events402a-402d(collectively referenced as402) of an object22transported by a carrying medium24, over a span of time. In one embodiment, the time interval between these events are assumed to not be a multiple of the frame time.

The support camera30may also capture start events404a-404d(collectively referenced as404) and stop events406a-406d(collectively referenced as406) in a sequence of image frames that it provides during the same span of time. The start and stop events404,406captured by of the support camera30may be the same start and stop events400,402captured by the main camera, but from a different viewpoint. Although they are corresponding events, the events are assigned different timestamps by the main camera10and the support camera30because the main camera10and the support camera30are out of temporal calibration.

In one embodiment, the autocalibration system100engages in iterative tentative alignments of the start and stop events400,402of the main camera10, to the start and stop events402,404of the support camera30. In one example, the autocalibration system takes the first two events400a,402aof the main camera, and matches it to the last two events404d,406dof the support camera30(or vice versa), so that the start event400aof the main camera10matches with the start event404dof the support camera, and the stop event402aof the main camera10matches with the stop event406dof the support camera30. In the case that the first tracked events of the first and support cameras10,30are of a different type (e.g., a “stop” event in the main camera10and a “start” event in the support camera30) and under the assumption that the types of events alternate between start and stop events, the autocalibration system10selects the second and third events instead of the first two. For example, the first tracked event for the main camera10may be a start of movement event while the first tracked event for the support camera30may be a stop of movement event.

In one embodiment, the autocalibration system100computes a temporary/tentative Δtibased on the match. As discussed above, the tentative calibration value for camera i(Δti) (where i is the second camera) may be an average of the temporal discrepancies between the timestamps of the first and second plurality of events, according to the following equation:

where n is the number of matched events, te,0is the timestamp of an event detected by the main camera10, and te,iis the timestamp of a matched event detected by the second camera.

As also discussed above, the autocalibration system100may further calculate a temporary error value E based on the temporary Δtiaccording to the below formula:

The error value may be saved in memory.

In one embodiment, the autocalibration system100attempts different alignments of the events of the support camera30that are earlier in time than the start and stop events404d,406dof the first iteration. In this regard, in performing a second alignment attempt, the process slides over to events earlier in time to attempt a match of the start event400awith start event404c, and a match of the stop event402awith stop event406d. In addition, additional matches are also attempted in the second iteration as more events become available for the support camera30for attempting the match. For example, a match may also be attempted between the start event400bof the main camera10and the start event404dof the support camera30, and between the stop event402bof the main camera10and the stop event406dof the support camera30.

In one embodiment, the autocalibration system100computes a second Δtibased on the second alignment attempt. In this regard, the computed Δtimay be an average of the discrepancies of the four matched events during the current iteration. The autocalibration system100may also calculate a new error value E using the newly computed Δti. The new error value may be compared against the stored error value for identifying a smaller one of the compared error values. Alternatively, the new error value is stored until all alignment possibilities have been attempted, and a minimal error value of the stored error values is selected as reflecting the correct alignment of events.

In one embodiment, the iterative alignment attempts continue by sliding the timeframe of the events of the main and support cameras to tentatively align the events until all alignment possibilities have been attempted. The error values corresponding to the alignment attempts are also computed based on the matches attempted during the alignment. The Δtiassociated with the minimal error value is identified as the autocalibration value for the support camera.

FIG.6is flow diagram of a process for subframe temporal autocalibration of a second imaging device (e.g. support camera30) to a first imaging device (e.g. main camera10) according to one embodiment. The process starts, and at block600, the autocalibration system100processes a first sequence of image frames transmitted by the first imaging device.

At block602, the autocalibration system100identifies a first plurality of events in the first sequence. For example, the event may be multiple start events (e.g. starts of motion of an object on the carrier medium24), multiple stop events (e.g. stops of motion of the object on the carrier medium24), and/or the like. The events may be identified using any algorithm conventional in the art such as, for example, an optical flow algorithm, keypoint detection algorithm, and/or the like.

At block604, the autocalibration system100processes a second sequence of image frames transmitted by a second imaging device (e.g. the support camera), and selects a second plurality of events as a match to the first plurality of events. In one embodiment, the selection of the second plurality of events is such that the discrepancy (e.g. an error measure) between the first plurality of events and the second plurality of events are minimized. In finding a match that minimizes the error measure, the autocalibration system100may be configured to try different matches in an iterative manner until all matches have been attempted, as discussed with reference toFIGS.4A-4B. Of course, any other minimization algorithm conventional in the art may also be employed.

At block606, the autocalibration system100identifies a first plurality of time values corresponding to the first plurality of events, and a second plurality of time values corresponding to the second plurality of events. The time values may be, for example, timestamps of the image frames capturing the events.

At block608, the autocalibration system100calculates a calibration value based on an average of the differences between the first plurality of time values and the second plurality of time values.

At block610, the autocalibration system100adjusts a timestamp of a third image frame provided by the second imaging device based on the calibration value. In this manner, the image frames provided by the second imaging device may be synchronized to the image frames of the first imaging device.

FIG.7is flow diagram of a process for temporal autocalibration of a second imaging device (e.g. support camera30) to a first imaging device (e.g. main camera10) in the presence of motion blur according to one embodiment. The process starts, and at block700, the autocalibration system100processes a first sequence of image frames transmitted by the first imaging device. In one embodiment, the autocalibration system100detects that one or more of the first sequence of images frames includes motion-blurred images due to movement of an object (e.g. object22) on the carrier medium24. In one embodiment, the autocalibration system100may assume presence of motion blur when a camera exposure time is bigger than a threshold value (e.g. bigger than 50% of the frame time).

At block702, the autocalibration system100computes a first plurality of blur kernels k for the first sequence of image frames. Each blur kernel may be an N×N convolution matrix that encodes the movement of the camera and/or objects in the scene. When the blur kernel k is applied to the latent sharp image I0, it recreates the captured blurred image I as follows:

where * indicates a convolution operation.

In one embodiment, the movements of the camera and/or objects are linear movements through the scene. In more detail, in some embodiments of the present disclosure, it is assumed that the camera remains stationary or fixed and the motion blur arises only from the (linear) movements of objects in the scene imaged by the camera. Given such linear movements, each blur kernel may appear as a linear stroke that may be converted to a 1D movement profile characterizing the movement of the object during the exposure time of the corresponding image frame. Accordingly, at block704, the autocalibration system100computes a first plurality of movement profiles for the first plurality of blur kernels.

At block706, the autocalibration system100computes a second plurality of blur kernels for a second sequence of image frames.

At block708, the autocalibration system100computes a second plurality of movement profiles for the second plurality of blur kernels.

At block710, the autocalibration system100matches the first plurality of movement profiles to the second plurality of movement profiles. In this regard, the autocalibration system100may combine/stack the movement profiles of each frame of the corresponding camera (see, e.g.FIG.10), and create a single, combined movement profile of the object for the camera over the time period of the sequence of image frames.

At block712, the autocalibration system100calculates a calibration value Δti so that the value minimizes an error value for all movement profiles (e.g., the difference between the movement profile of the object as captured by the main camera and each of the movement profiles of the object as captured by the support cameras i) combined:

At block714, the autocalibration system100adjusts a timestamp of a third image frame provided by the second imaging device based on the calibration value. In this manner, the image frames provided by the second imaging device may be synchronized to the image frames of the first imaging device.

FIG.8is a more detailed flow diagram of the steps in blocks704and708for computing a one-dimensional (1D) movement profile based on a blur kernel according to one embodiment. The process starts, and at block800, the autocalibration module100detects one or more linear strokes in the kernel.

At block802, the autocalibration module100rotates the one or more linear strokes, as needed, so that they lie on a horizontal line. In particular, the different cameras capture images of the scene from different viewpoints. Based on the poses (position and orientation) of the cameras, the same moving object may appear to be moving in different directions with respect to the image frames captured by the different cameras. For example, a main camera located level with the moving object may detect motion blur due to the object moving left to right from the viewpoint of the main camera, while a support camera located above (and looking down at) the moving object may detect motion blue due to the object moving top to bottom from the viewpoint of the support camera. Therefore, the blur kernel computed from the image frame of the main camera may be horizontal line due to the left-to-right motion, whereas the blur kernel computed from the image frame captured by the support camera may be a vertical line due to the top-to-bottom motion. In a similar way, a third camera having an above-side view of the object may have a viewpoint where the object moves diagonally through its field of view, and the motion blur kernel may therefore be a diagonal line along the direction of motion of the object through the image frame. Therefore, in one embodiment, the autocalibration module100rotates the linear strokes of the motion blur kernels corresponding to the different cameras to a common direction (e.g., a horizontal direction) to convert the 2D motion profiles represented by the motion blur kernels into 1D motion profiles. In one embedment, the length of the linear stroke corresponds to the speed of the linear motion of the object during a frame. For example, a stopped object has no motion blur and therefore the motion blur kernel will have zero length, and a fast moving object will have a large amount of motion blur and therefore the motion blur kernel will be long. A vector of values (e.g. [0.3, 0.4, 0.4, 0.9]) is identified based on the lengths of the one or more linear strokes.

At block804, a function x(t) is identified by inverting the vector of values. In this regard, the vector is normalized, such as by dividing the vector values by the sum of all the values of the vector (e.g. 2 for the above example). In the above example of vector values, the result of the normalization is [0.15, 0.2, 0.2, 0.45]. The normalization may be performed because the different viewpoints of the different cameras may cause the lengths of the strokes to be different (e.g., given two cameras having the same field of view, a camera that is closer to the object will observe more motion blur than a camera farther away from the object). The exposure time for the frame is multiplied by each vector value representing a normalized speed of the object during each frame. For example, if the exposure time is 100 ms, the values in the example above are: [15, 20, 20, 45]. The function x(t) may define the movement profile in the following way:

It should be understood that the sequence of steps of the process in any ofFIGS.5-8is not fixed, but can be modified, changed in order, performed differently, performed sequentially, concurrently, or simultaneously, or altered into any desired sequence, as recognized by a person of skill in the art.

FIGS.9A-9Bare pictures of exemplary motion blur kernels according to one embodiment. The motion blur kernel ofFIG.9Adepicts a linear stroke900generated by an object moving linearly in a diagonal direction.FIG.9Bdepicts the motion blur kernel ofFIG.9A, but with the linear stroke900rotated to be in a horizontal direction.

FIG.10is a graph of an example 1D movement profile according to one embodiment. In the example ofFIG.10, the movement profile is defined by a step function.

Once the support cameras30are temporally calibrated to the main camera10, the image frames provided by the cameras10,30may be used for estimating the six-degree-of-freedom (6-DoF) poses of objects in a scene, and/or the moving direction and speed of the objects. Such information may be useful in various applications such as robotics. For example, such information may be provided to a robot controller for improving situational awareness and enabling the robot controller to interact appropriately with the environment, in accordance the particular tasks assigned to the robot. As a further example, autonomously navigating robots or vehicles may maintain information about the poses of objects in a scene in order to assist with navigation around those objects in order to predict trajectories and to avoid collisions with those objects. As another example, in the case of manufacturing, pose estimation may be used by robotic systems to manipulate the workpieces and place and/or attach components to those workpieces.