Patent Description:
The following disclosure relates to an electronic device and method with independent time point management.

A camera may not implement precise time control when being used as a simple image capturing device, as the camera may perform computations through sequential processing according to the performance of a computing device through simple scheduling and the camera may perform operations in a manner of outputting results when the results are obtained. Such a device is illustrated in US patent application <CIT>. On the other hand, a system implementing strict time control, such as an aircraft or a robot, may perform computations ensuring an accurate processing time using a real-time operating system (RTOS).

Inexpensive sensor systems may be implemented in general consumer electronic devices such as cell phones. Sensor fusion operations may be used in various devices including control systems having autonomous driving. In these applications, it may be difficult to use a system such as the aforementioned RTOS due to cost and convenience. For example, a simultaneous localization and mapping (SLAM) algorithm in a robot or a vehicle may require strict time control for accurate performance, but adverse to such requirement, the robot or vehicle may include a combination of sensors not temporally synchronized with a typical general-purpose operating system (OS).

In accordance with its abstract, patent application publication <CIT> describes an image processing device comprising: a camera capturing an image feed of a scene; a sensor obtaining sensor data, representative for an orientation of the image processing device; a first estimating unit estimating a position and direction of the camera at a first moment in time, based on a first captured image of the scene; a renderer rendering a virtual object at a second moment in time; a second estimating unit estimating a position and direction of the camera at the second moment in time, based on the estimated position and direction of the camera at the first moment in time and sensor data obtained at the first and second moments, wherein the renderer renders the virtual object based on the estimated position and direction of the camera at the second moment in time; and a display displaying the rendered virtual object in registration with the image feed.

Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same or like elements, features, and structures.

Also, descriptions of features that are known, after an understanding of the disclosure of this application, may be omitted for increased clarity and conciseness.

Although terms, such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms.

The articles "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term "and/or" includes any one and any combination of any two or more of the associated listed items. The use of the term "may" herein with respect to an example or embodiment (for example, as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the examples with reference to the accompanying drawings, like reference numerals refer to like components and a repeated description related thereto will be omitted.

<FIG> illustrates an example of an electronic device including a timekeeping module.

Referring to <FIG>, an electronic device <NUM> may include a sensor <NUM> (e.g., one or more sensors), a timekeeping module <NUM>, a processor <NUM> (e.g., one or more processors), an output module <NUM> (e.g., a user interface including a display), and a memory <NUM> (e.g., one or more memories). For example, the electronic device <NUM> may determine localization of the electronic device <NUM> based on sensing data generated through a plurality of sensors. The localization of the electronic device <NUM> may include either one or both of a pose and a location of the electronic device <NUM>. The electronic device <NUM> may perform a sensor fusion operation on the sensing data generated through the plurality of sensors. The electronic device <NUM> may perform software synchronization of the plurality of sensors to perform the sensor fusion operation. For example, the software synchronization may be accurately monitoring times of sensing by the sensors and a difference between the times of sensing, in the sensor fusion operation. The electronic device <NUM> may determine the localization of the electronic device <NUM> by performing the sensor fusion operation in view of the difference between the times of sensing, without forcibly matching measurement times of sensing by the sensors.

The sensor <NUM> may include a plurality of sensors. The plurality of sensors may generate sensing data for localization through sensing. For example, one or more of the plurality of sensors may be a heterogeneous sensor that generates different types of sensing data from that of another sensor of the plurality of sensors. Of the plurality of sensors, a reference sensor may be a sensor that generates reference sensing data that serves as a reference for localization. The other sensor may be a sensor that generates other sensing data used to perform localization in addition to the reference sensing data. The localization may be determined based on the reference sensing data and the other sensing data with respect to a sensing time point of the reference sensing data (e.g., a reference sensing time point).

For example, the reference sensor may include a vision sensor. The vision sensor may be a sensor for performing vision-based sensing and may include, for example, a camera sensor, a radar sensor, a lidar sensor, and/or an ultrasonic sensor, as a non-limiting example. The camera sensor may capture a camera image. The radar sensor may sense radar data. The lidar sensor may sense a lidar image. However, examples are not limited thereto. The vision sensor may include a depth camera sensor, a stereo camera sensor, a multi-camera sensor, and/or a single-camera sensor.

The other sensor may include any one or any combination of any two or more of an inertial measurement unit (IMU) configured to sense an inertia of the electronic device <NUM>, an active sensor configured to perform sensing in response to receiving a trigger input, and a passive sensor configured to perform sensing at predetermined intervals. The IMU may be a sensor for measuring an inertia applied thereto and may sense a <NUM>-axis acceleration and/or a <NUM>-axis angular velocity. The active sensor may perform sensing in response to receiving a trigger input and thus, may generate sensing data irregularly based on the received trigger input. The passive sensor may perform sensing at intervals according to its clock and thus, may generate sensing data regularly. For reference, the timekeeping module <NUM> may control a timing of a trigger input for the active sensor and apply the trigger input to the active sensor.

Herein, an example of the reference sensor being a vision sensor (e.g., a camera sensor) and the other sensor being an IMU is described only for purposes of ease of explanation. However, the reference sensor and the other sensor are not limited thereto.

The timekeeping module <NUM> may operate at a separate clock rate from and independently of the processor <NUM> (non-limiting examples of which will be described later), and generate, record, calibrate, store, and manage time information related to a target task in the electronic device <NUM>. The time information related to the target task may include sensing time points determined for the respective sensing data, a time difference between the sensing data (non-limiting examples of which will be described later), and a task latency (non-limiting examples of which will be described later). The timekeeping module <NUM> may store accurate time information of the respective sensing data. The timekeeping module <NUM> may provide the stored time information of the sensing data to a processor (e.g., the internal application processor <NUM> or a processor of an external device) during an operation for the target task.

The target task may be a task to be performed with respect to the determined localization, and may include one or more operations and/or computations to be performed at the localization determined with respect to the electronic device <NUM>. For example, when the electronic device <NUM> is a device that provides augmented reality (AR) and/or virtual reality (VR), the target task may include an operation of rendering an image for AR and/or VR (e.g., an image including virtual content) to be provided to a user at the localization determined with respect to the electronic device <NUM>. As another example, when the electronic device <NUM> is implemented as a robot, the target task may include a series of operations (e.g., recognizing a periphery and/or outputting recognition results) for performing an action (e.g., serving an article, guiding with voice, and/or interacting with a peripheral object such as opening a door) to be performed at a localization determined with respect to the robot. As still another example, when the electronic device <NUM> is mounted on, or is, a vehicle (e.g., a car, an aircraft, and/or a watercraft), the target task may include an operation (e.g., recognizing a periphery and steering and/or controlling a speed according to recognition results) for driving and/or flying at a localization determined with respect to the electronic device <NUM>. However, the examples of the target task are not limited thereto.

The timekeeping module <NUM> may record sensing time points (e.g., timestamps) for respective sensors. For example, the timekeeping module <NUM> may determine a sensing time point of reference sensing data (e.g., a reference sensing time point) generated through sensing by a reference sensor and a sensing time point of other sensing data of another sensor (e.g., another sensing time point), among the plurality of sensors, based on a clock rate of the timekeeping module <NUM>. The timekeeping module <NUM> may separately record and manage a timestamp corresponding to the reference sensing time point and a timestamp corresponding to the other sensing time point. For reference, the timekeeping module <NUM> may include a clock generation circuit that operates at a clock rate (e.g., a clock frequency) independent of a clock (or clock rate thereof) of the processor <NUM>. The clock rate of the timekeeping module <NUM> may be faster than the sensing frequencies of the plurality of sensors. The timekeeping module <NUM> of one or more embodiments may determine, independently of the processor <NUM>, the respective sensing time points at a clock rate faster than the sensing frequencies of the sensors, thereby more accurately recording and managing the sensing time points. For example, the clock rate of the timekeeping module <NUM> may be more than <NUM> times faster than the fastest sensing frequency of the plurality of sensors. However, the example of the clock rate is not limited thereto.

The timekeeping module <NUM> may also include a memory for storing time information related to the target task and timing calibration information for timing calibration of the time information. The timekeeping module <NUM> may calculate (e.g., determine) more precise sensing time points by calibrating the sensing time points using the timing calibration information. A non-limiting example of the timing calibration information will be described below with reference to <FIG>.

The timekeeping module <NUM> may calculate a time difference between the sensing time point of the reference sensing data and the sensing time point of the other sensing data. For example, the timekeeping module <NUM> may calculate a time difference from the other sensing time point based on the reference sensing time point, e.g., because the electronic device <NUM> may obtain (e.g., determine) localization based on the reference sensing time point, a non-limiting example of which will be described below.

The timekeeping module <NUM> may determine a task latency from the sensing time point of the reference sensing data to a task complete time point based on the clock rate of the timekeeping module <NUM>. The task latency may be a latency time taken from the reference sensing time point to the task complete time point. The timekeeping module <NUM> may record the task complete time point (e.g., a time point when rendering is completed), and calculate a time from a last reference time point immediately before a task is initiated to the task complete time point as the task latency. For example, when the target task is to render an image for AR, the task complete time point may be a time point when rendering is completed. The task latency may include a rendering latency. The timekeeping module <NUM> may determine a rendering latency from the sensing time point of the reference sensing data to a time point when rendering of an image for AR is completed.

The processor <NUM> may obtain the localization of the electronic device <NUM> determined for the sensing time point of the reference sensing data based on the calculated time difference, the reference sensing data, and the other sensing data. The processor <NUM> may obtain a task result processed with respect to the localization. When the target task is to render an image for AR, the task result may include a rendered image to be provided to a user at the localization at the reference sensing time point. For example, a field of view corresponding to a view direction of the user may be determined based on a location and an orientation of the electronic device <NUM> included in the localization. The task result may include an AR image having virtual content to be provided to or in the field of view corresponding to the view direction.

The processor <NUM> may correct, based on the task latency, a task result processed according to the localization of the electronic device <NUM> determined for the sensing time point of the reference sensing data based on the calculated time difference, the reference sensing data, and the other sensing data. The processor <NUM> may compensate for changed localization (e.g., a movement and a direction change) during the task after the reference sensing time point in the task result. For example, when the target task is to render an image for AR, the processor <NUM> may generate an output image by correcting an image rendered according to the localization of the electronic device determined for the sensing time point of the reference sensing data based on the rendering latency.

In an example, the processor <NUM> may be implemented as an application processor including a main processor and a sub-processor as one or more processors. As described below with reference to <FIG>, an operation of determining the localization of the electronic device <NUM> and an operation of processing the task (e.g., rendering) corresponding to the determined localization may be performed by the application processor. However, examples are not limited thereto. As another example, as described below with reference to <FIG>, the operation of determining the localization of the electronic device <NUM> and the operation of processing the task corresponding to the determined localization may be performed by an external device, and the processor <NUM> of the electronic device <NUM> may receive and obtain the determined localization and the task result according to the localization from the external device.

The hardware including output module <NUM> may output a corrected task result based on the task latency. The output module <NUM> may output any one or any combination of any two or more of visual information, auditory information, and tactile information according to the form of the task result. For example, when the target task is to render an image for AR, the output module <NUM> may include a display for displaying an output image generated to provide AR. The output module <NUM> may generate a user interface and render the image using the generated user interface, for example.

The memory <NUM> may store instructions that, when executed by the processor <NUM>, configure the processor <NUM> to perform any one, any combination, or all operations of the processor <NUM>. The memory <NUM> may include, for example, a random-access memory (RAM), a dynamic RAM (DRAM), a static RAM (SRAM), or other types of nonvolatile memory that are known in the related technical field.

The electronic device <NUM> of one or more embodiments may obtain the localization (e.g., the pose and the location) with respect to, or corresponding to, the reference sensing time point based on sensing data for respective multiple sensors when actually calculating the target task, thereby performing more precise motion estimation. Further, the electronic device <NUM> of one or more embodiments may provide the user with a result (e.g., a corrected task result) of compensating for the localization change from the localization determination at the reference sensing time point to a time point when the task result is outputted, thereby providing the user with a more accurate result.

As described above, the electronic device <NUM> may include the separate timekeeping module <NUM> independent of the application processor. The timekeeping module <NUM> may include a clock generation circuit that operates with a relatively accurate clock (e.g., a more accurate clock than a clock of one or more sensors of the sensor <NUM>). The timekeeping module <NUM> may monitor operating times of the sensors, times taken for the operations (e.g., sensing) of the sensors, and a time taken for an operation for the target task by the processor <NUM> and/or the external device all. Accordingly, the timekeeping module <NUM> may calculate an end-to-end latency (e.g., the task latency) from a sensing time point of a sensor (e.g., a timestamp recorded for sensing) based on the monitored time information. Accordingly, the electronic device <NUM> of one or more embodiments implemented as a real-time system (e.g., an AR device, a simultaneous localization and mapping (SLAM) device, and various mechanical control devices) measuring or controlling a motion may minimize an error caused by an end-to-end latency occurring in an end-to-end operation.

Sensor fusion may implement strict time synchronization of sensors included in hardware. As described above, the electronic device <NUM> of one or more embodiments may record and manage sensing time points for respective sensors through the timekeeping module <NUM>, thereby applying sensor fusion even to sensors having different clock rates. For example, when outputs of the respective sensors are generated according to individual clocks of the respective sensors, output intervals of the heterogeneous sensors may be asynchronous. In addition, commercial sensors may output and transmit sensor data periodically according to their own clock rates, and clock rates generated by clock circuits mounted in the individual sensors may show significant errors.

In addition to the errors or differences of the clock rates for the respective sensors, for electromagnetic or communication reasons, the intervals for sampling sensing data from the sensors may vary. For example, in the case of low-cost IMUs commonly used in cell phones, clock rates may vary up to approximately <NUM>% depending on a situation. According to a comparative example, the electronic device <NUM> of one or more embodiments may calculate the velocity or pose by integrating an angular velocity or acceleration sensed through the IMU. When the integral of the angular velocity or acceleration is used, a change in time reference in the estimation using the IMU data may be linearly reflected in a measurement error. In the case of camera sensors, shutter speeds and data processing rates may vary for the respective camera sensors even when an image is captured at the same frames per second (fps). Accordingly, data having different latencies and different sampling intervals may be generated for each frame image measurement. This may make it difficult for a typical electronic device to specify data measured at the same time as data generated through a first camera sensor from among data generated by first camera sensors and having different latencies. For reference, a portion of the sensors (e.g., a camera sensor) may be designed or configured to receive a trigger input, but it may be difficult to control a trigger input timing.

The electronic device <NUM> of one or more embodiments may record the accurate sensing time points of the sensing data based on the clock rate of the timekeeping module <NUM> despite the errors described above and thus, may accurately perform a localization operation (e.g., SLAM) based on sensor fusion. The electronic device <NUM> of one or more embodiments may even record the sensing time points for the respective sensors regardless of the presence or absence of a trigger, and thus, the diversity and flexibility in selecting a sensor for the electronic device <NUM> may be secured in a development stage.

Furthermore, the electronic device <NUM> of one or more embodiments may also help toward preventing, or prevent, a latency issue due to a computation time occurring in AR. The processor <NUM> may accurately manage task scheduling results of general-purpose operating systems (OS) and variable latencies caused by a difference in amount of computation in each situation, whereby the electronic device <NUM> of one or more embodiments may reduce or minimize errors caused by an end-to-end latency.

<FIG> and <FIG> illustrate examples of performing an operation for a task internally by an electronic device.

In <FIG>, an example in which sensors <NUM> (e.g., the sensor <NUM> of <FIG> as a non-limiting example) include a first sensor and a second sensor, a timekeeping module <NUM> (e.g., the timekeeping module <NUM> of <FIG> as a non-limiting example) includes a timekeeping processor <NUM> and a memory <NUM>, and an application processor <NUM> (e.g., the processor <NUM> of <FIG> as a non-limiting example) includes a main processor <NUM> and a sub-processor <NUM> is shown. The main processor may be a central processing unit (CPU), and the sub-processor may be a graphics processing unit (GPU). However, examples are not limited thereto. The output module <NUM> of <FIG> may be implemented as a display <NUM>. Hereinafter, a non-limiting example in which a target task is to provide an AR image for AR is mainly described.

When the target task is to generate an AR image for AR, a main operation of an electronic device <NUM> may include an operation of determining localization (e.g., SLAM vision), an operation of modeling a surrounding environment of the electronic device <NUM> in three-dimensional (3D) geometry (e.g., interpreting and/or modeling a physical object, a person, and a background in 3D coordinates), and an operation of generating and rendering an AR image including virtual content (e.g., a virtual object). The operation of determining the localization and the operation of modeling the surrounding environment in 3D geometry may be performed by the main processor <NUM>. Rendering the AR image may be performed by the sub-processor <NUM>. For example, the main processor <NUM>, of one or more processors of the application processor <NUM>, may determine localization of the electronic device <NUM> based on a time difference calculated for a sensing time point of reference sensing data (e.g., first sensing data) (e.g., a difference between sensing time points from first sensing to second sensing), the reference sensing data (e.g., the first sensing data), and other sensing data (e.g., second sensing data). The sub-processor <NUM>, of the one or more processors, may generate a rendered image as a task result for the localization. An example in which the electronic device <NUM> including the main processor <NUM> and the sub-processor <NUM> autonomously performs a task-related operation will be described below.

For example, in operation <NUM>, the first sensor <NUM> may generate first sensing data by performing first sensing. The timekeeping module <NUM> may receive a sensing trigger of the first sensing in response to the occurrence of the first sensing (e.g., the first sensor <NUM> may generate and transmit the sensing trigger of the first sensing to the timekeeping module <NUM>, in response to the performing of the first sensing). The sensing trigger may be a signal from each sensor to inform the timekeeping module <NUM> of the generation of sensing data, and may, for example, indicate that the corresponding sensor has completed capturing an image and/or has sensed acceleration. In operation <NUM>, the timekeeping module <NUM> may determine and record, for the first sensing data, a timestamp of a clock timing at which the sensing trigger is received. Non-liming examples of recording the clock timing and the timestamp will be described below with reference to <FIG> and <FIG>. In operation <NUM>, the second sensor <NUM> may generate second sensing data by performing second sensing. The timekeeping module <NUM> may receive a sensing trigger of the second sensing in response to the occurrence of the second sensing. In operation <NUM>, the timekeeping module <NUM> may determine and record a timestamp for the second sensing data based on a clock rate of the timekeeping module <NUM>. For reference, the timekeeping module <NUM> may determine the above-described sensing time points based on the clock rate of the timekeeping processor <NUM>, and store the determined sensing time points in the memory <NUM>.

In operations <NUM> and <NUM>, the first sensor <NUM> and the second sensor <NUM> may each transmit sensing data to the main processor <NUM>. For reference, as an example, only the first sensor <NUM> and the second sensor <NUM> are shown in <FIG> and <FIG> for ease of explanation. However, examples are not limited thereto. In the present specification, the sensors <NUM> may include n sensors, where n may be an integer greater than or equal to "<NUM>".

For reference, the timekeeping module <NUM> may generate a trigger input and provide the trigger input to a sensor capable of receiving (or configured to receive) the trigger input (e.g., an active sensor), of the first sensor <NUM> and the second sensor <NUM>. The sensor configured to receive the trigger input may perform sensing in response to receiving the trigger input. Accordingly, a timing of the trigger input may also be determined based on the clock rate of the timekeeping module <NUM>. In the present specification, a camera sensor is described as an example of the first sensor <NUM> (e.g., a reference sensor). However, examples are not limited thereto. Other vision sensors such as a radar sensor and a lidar sensor may also be operated as the reference sensor in the same manner as the camera sensor.

In operation <NUM>, the timekeeping module <NUM> may calculate a time difference for each sensing data. For example, the timekeeping module <NUM> may calculate a time difference between sensing data based on a last timestamp of the first sensor <NUM>. The timekeeping module <NUM> may calculate and record a time difference from a sensing time point of other sensing data (e.g., the timestamp for the second sensing data) based on a last sensing time point of first sensing data (e.g., the timestamp for the first sensing data) collected until the localization is determined, that is, a latest sensing time point (e.g., a latest reference time point).

In operation <NUM>, the timekeeping module <NUM> may transmit time-related information to the main processor <NUM>. For example, the timekeeping module <NUM> may transmit, to the main processor <NUM>, time information monitored for the sensing data (e.g., the timestamp for each sensing data and the calculated time difference between the sensing data). For reference, the timekeeping module <NUM> may transmit all time information at once. However, examples are not limited thereto. While the main processor <NUM> and the sub-processor <NUM> perform operations for a task, a portion of the time information requested for each operation may be selectively provided to the main processor <NUM> and/or the sub-processor <NUM>.

In operation <NUM>, the main processor <NUM> of the electronic device <NUM> may determine the localization of the device at the reference sensing time point (e.g., the timestamp for the first sensing data) of the first sensor <NUM> (e.g., the reference sensor) based on the time information described above. The main processor <NUM> may determine the localization of the electronic device <NUM> based on a time point that is temporally closest to a localization time point (e.g., the latest reference sensing time point), of the reference sensing time points. The localization time point may be a time point when the electronic device <NUM> initiates an operation for determining the localization. In operation <NUM>, the main processor <NUM> may transmit the determined localization (e.g., AR device localization) to the sub-processor <NUM>.

In operation <NUM>, the sub-processor <NUM> may render an image for the reference sensing time point (e.g., the timestamp for the first sensing data) based on the determined localization. For example, the sub-processor <NUM> may render and generate an AR image corresponding to a field of view in the localization of the electronic device <NUM> at the reference sensing time point. In operation <NUM>, the sub-processor <NUM> may transmit a rendering complete trigger to the timekeeping module <NUM>. For example, in response to the rendered image being generated, the sub-processor <NUM> may notify the timekeeping module <NUM> of the completion of rendering. In operation <NUM>, the timekeeping module <NUM> may determine a rendering complete time point based on a clock rate of the timekeeping module <NUM>.

In operation <NUM>, the timekeeping module <NUM> may calculate a rendering latency from the latest reference sensing time point (e.g., the timestamp for the first sensing data) to the rendering complete time point. For example, the timekeeping module <NUM> may calculate the rendering latency by subtracting the latest reference sensing time point from the rendering complete time point. In operation <NUM>, the timekeeping module <NUM> may transmit the calculated rendering latency to the main processor <NUM>. In operation <NUM>, the main processor <NUM> may generate an output image by compensating the rendered image based on the calculated rendering latency. For example, in operation <NUM>, the main processor <NUM> may perform a latency compensation operation <NUM>. The latency compensation operation <NUM> may include warping based on any one or any combination of any two or more of a rotation transform, a translation, and a scale change corresponding to a localization change of the electronic device <NUM> during a task latency from the sensing time point of the reference sensing data to a task complete time point. For reference, it is described that the latency compensation operation <NUM> is performed by one of the processors of the application processor <NUM>, but is not limited thereto, and may be performed by another separate processor (e.g., a dedicated processor for latency compensation) in the electronic device. In operation <NUM>, the main processor <NUM> may transmit a corrected image to the display <NUM>. In operation <NUM>, the display <NUM> may display the generated output image.

The electronic device <NUM> may match coordinates of a real object and coordinates of a virtual object at a low latency in the field of AR and mixed reality (MR). The electronic device <NUM> of one or more embodiments may implement a low latency in mobile computing and/or edge computing, thereby preventing a coordinate mismatch between a real object and a virtual object due to a delay. To prevent the coordinate mismatch, the timekeeping module <NUM> of one or more embodiments operating with a clock separate from those of the main processor <NUM> and the sub-processor <NUM> in the electronic device <NUM> may separately manage time information related to localization and provide the main processor <NUM> with time information used for an individual operation.

<FIG> and <FIG> illustrate examples of determining a timestamp by a timekeeping module.

A timekeeping module (e.g., either one or both the timekeeping module <NUM> and the timekeeping module <NUM>) may determine a sensing time point of sensing data generated based on sensing by all sensors based on a clock rate of a timekeeping processor (e.g., the timekeeping processor <NUM>). The timekeeping module may record a timestamp determined based on the clock rate of the timekeeping module for each generated sensing data. For example, k-th sensing data may be generated based on k-th sensing by one sensor. The timekeeping module may receive a sensing trigger indicating the generation of the k-th sensing data from the sensor. Here, k may be an integer greater than or equal to "<NUM>". The timekeeping module may determine a clock timing immediately after the sensing trigger is received (e.g., a next clock timing generated once the sensing trigger is received) to be a sensing time point for the corresponding sensing data. The sensing time point may be recorded as a timestamp. All sensors included in the electronic device have their own clock circuits and may sense, generate, and transmit data at predetermined intervals (e.g., intervals according to clock rates determined by individual clock circuits). When outputs of the respective sensors are generated according to individual clocks, output intervals of the respective sensors may be asynchronous.

<FIG> illustrates an example operation of determining a sensing time point of a passive sensor (e.g., the second sensor <NUM> as a non-limiting example). An IMU will be described as an example of the passive sensor. The IMU may be a sensor that measures inertia, and may sense, for example, a <NUM>-axis acceleration and/or a <NUM>-axis angular velocity. The IMU may operate with an independent clock <NUM>, and the IMU may operate at a faster rate than a camera sensor (e.g., the first sensor <NUM> as a non-limiting example). When the IMU is a low-price sensor, the IMU may operate at a predetermined clock rate without a trigger. The IMU may generate IMU data as the sensing data by sensing the <NUM>-axis acceleration and/or the <NUM>-axis angular velocity applied to the electronic device at every interval according to its own clock rate. When the IMU is the low-price sensor, a clock error of the IMU may occur approximately up to <NUM>%. In response to receiving a sensing trigger <NUM> of the IMU data from the IMU, the timekeeping module may store a timestamp (e.g., the timestamp for the second sensing data) indicating a latest (e.g., a next subsequent) timing <NUM> updated according to the clock rate of the timekeeping processor after the sensing trigger <NUM> as a sensing time point <NUM> of the IMU data.

For reference, when a clock <NUM> of the timekeeping module is sufficiently fast, a difference <NUM> between an actual time of the generation of the IMU data and the sensing time point indicated by the timestamp may be trivial. For example, when the clock <NUM> of the timekeeping module is faster than the clock <NUM> of the IMU, the error may be ignored. For example, the clock rate of the timekeeping module may be more than <NUM> times faster than the fastest sampling rate of the plurality of sensors, and thus the error may be ignored. The timekeeping module may have a clock rate better than the error level of the IMU.

<FIG> illustrates an operation of determining a sensing time point of a camera sensor among vision sensors.

According to an example, a reference sensor (e.g., the first sensor <NUM> as a non-limiting example) of an electronic device may include a camera sensor that captures a camera image, among vision sensors. The camera sensor may generate image data (e.g., camera data) by capturing a scene. The camera sensor may perform continuous shooting at predetermined frames per second (FPS). In <FIG>, a clock <NUM> of a timekeeping module and a sensing time axis <NUM> of the camera sensor are shown.

Similar to the IMU described above with reference to <FIG>, the timekeeping module may use a latest clock time point (e.g., the timestamp for the first sensing data) for the camera sensor. For example, a sensing time point of camera sensing data of the camera sensor may be determined to be a timestamp indicating a clock timing <NUM> of the timekeeping module after a sensing trigger <NUM> indicating that a k-th image is obtained. For example, in response to receiving the sensing trigger <NUM> of the camera sensing data from the camera sensor, the timekeeping module may store a timestamp indicating a latest time updated according to its own clock rate after receiving the sensing trigger <NUM> as the sensing time point of the camera sensing data. Accordingly, the timekeeping module may record a sensing time point of the k-th image data based on the clock timing at the time point when reading of the camera data is finished. The timekeeping module may be designed to have a clock rate sufficiently faster than sampling rates (e.g., data generation frequencies) of the sensors including the camera sensor.

However, examples are not limited thereto. Additionally, the sensing time point of the camera data may be calibrated. For example, the timekeeping module may calibrate the sensing time point of the camera sensor based on either one or both of an exposure time <NUM> and a read time <NUM> of the camera image. For example, a generation time point <NUM> of the camera data may be treated as a time point <NUM> corresponding to the median of the exposure time <NUM>. On the other hand, the sensing trigger of the camera sensing data may occur after the exposure time <NUM> and the read time <NUM> elapse and reading is completed. In other words, there may occur a difference between the generation time point <NUM> of the camera sensing data and the read time point of the camera sensing data. The difference between the generation time point <NUM> of the camera sensing data and the read time point of the camera sensing data may be calculated as a calibration time <NUM> by the timekeeping module. According to the example shown in <FIG>, the difference between the generation time point <NUM> and the read time point of the camera sensing data may be expressed as Equation <NUM> below, for example.

In Equation <NUM> above, the generation time point <NUM> of the camera sensing data may be assumed to be a time point corresponding to half of the exposure time. The timekeeping module may calibrate a sensing time point of camera data when a read time and an exposure time are given.

For example, the camera sensor may provide an exposure value representing the exposure time <NUM> to the timekeeping module. The timekeeping module may store the exposure time <NUM> in a memory (e.g., the memory <NUM>). The timekeeping module may determine a calibrated sensing time point for the camera sensing data by subtracting a time corresponding to a half the exposure time <NUM> from the clock timing <NUM> immediately after the sensing trigger <NUM> occurs.

Furthermore, it may be assumed that the read time <NUM> is constant for every capturing by the camera sensor. The timekeeping module may store in advance the read time <NUM> measured for the camera sensor in the memory. The timekeeping module may determine a sensing time point <NUM> more precisely calibrated for the camera sensing data by subtracting the calibration time <NUM> (e.g., the sum of the read time <NUM> and a half the exposure time <NUM>) from the clock timing <NUM> immediately after the sensing trigger <NUM> occurs.

<FIG> illustrates timings for determining localization, rendering, and displaying an output image by an electronic device according to the disclosure.

A timekeeping module <NUM> of an electronic device determines sensing time points for sensing data of sensors <NUM>. For example, the timekeeping module <NUM> may determine sensing time points respectively for a camera sensor <NUM>, an IMU <NUM>, a passive sensor <NUM>, and an active sensor <NUM>. The electronic device may correspond to the electronic device of any one, any combination, or all of <FIG>, as a non-limiting example.

In operation <NUM>, the timekeeping module <NUM> determines a localization timestamp when localization by a main processor <NUM> is initiated. The localization timestamp may be determined based on reference sensing data at a latest reference sensing time point of the sensing time points (e.g., reference sensing time points) of reference sensing data collected until a time point when the localization is performed. A sensing time point of a vision sensor may be a reference sensing time point of the localization. The reference sensing data may be camera sensing data sensed by the camera sensor <NUM> as the vision sensor, as described above.

In operation <NUM>, the main processor <NUM> (e.g., the main processor <NUM> of <FIG> as a non-limiting example) determines the localization based on the localization timestamp. For example, the main processor <NUM> may determine the localization (e.g., a pose and a location) of the electronic device at the localization timestamp, based on the respective sensing data and sensing time points of the sensors <NUM> on the basis of the time point corresponding to the localization timestamp.

In operation <NUM>, in response to determining the localization with respect to the localization timestamp, the main processor <NUM> instructs initiation of a task (rendering). The main processor <NUM> instruct a sub-processor <NUM> (e.g., the sub-processor <NUM> of <FIG>) to initiate rendering.

In operation <NUM>, the timekeeping module <NUM> determines and records a time point indicating the completion of rendering based on a clock rate of a timekeeping processor (a timekeeping processor of the timekeeping module <NUM>). The timekeeping module <NUM> calculates a rendering latency, which is a time taken from the localization timestamp to the completion of rendering. Accordingly, the timekeeping module <NUM> may calculate the rendering latency including a rendering initiation time point since a time point corresponding to the localization timestamp determined in operation <NUM>, and a time from the rendering initiation time point to a rendering complete time point.

In operation <NUM>, the main processor <NUM> warps a rendered image based on the rendering latency. The main processor <NUM> predicts a localization change of the electronic device from the localization timestamp to the rendering complete time point, and perform warping corresponding to the predicted localization change. The main processor <NUM> may generate an output image by warping the rendered image. The warping may be an operation of transforming a coordinate system of pixels included in the rendered image, a non-limiting example of which will be described below with reference to <FIG>.

In operation <NUM>, a display <NUM> (e.g., the display <NUM> of <FIG>) may display the output image. For example, the display <NUM> may provide the output image to a user by forming an image on a transparent or translucent material. The output image may include a pair of a first image (e.g., a left image) and a second image (e.g., a right image), and the display <NUM> may provide the first image and the second image toward a portion corresponding to the left eye and a portion corresponding to the right eye of the user, respectively, thereby providing a stereoscopic image having depth to the user.

As described above, the electronic device may monitor and record a time point when data computation is finished (e.g., an instant when the rendered image is transmitted to the display) through the timekeeping module <NUM>. Accordingly, the electronic device may directly calculate an end-to-end latency (delay) from the sensing by the sensor to the completion of rendering. The electronic device of one or more embodiments may reduce an error caused by the rendering latency described in <FIG> as well as an error caused by the difference between the sensing time points of the sensors <NUM> described in <FIG>. Accordingly, the electronic device of one or more embodiments may greatly reduce recognition and rendering errors in a situation in which the electronic device is moving.

<FIG> and <FIG> illustrate examples of requesting an operation for a task from an external server by an electronic device.

In an electronic device <NUM> shown in <FIG>, sensors <NUM> including a first sensor <NUM> and a second sensor <NUM>, a timekeeping module <NUM> including a timekeeping processor <NUM> and a memory <NUM>, and a display <NUM> may operate similarly as described above in <FIG>, and thus, a description thereof will be omitted. For example, the electronic device <NUM> may correspond to the electronic device of any one, any combination, or all of <FIG>, as a non-limiting example. In an example, unlike the electronic device <NUM> shown in <FIG>, the electronic device <NUM> of <FIG> may request a series of computations for a target task from a separate computing device <NUM> (e.g., an external device). The separate computing device <NUM> may be implemented as an edge computing device or a cloud server having a main processor <NUM> and a sub-processor <NUM> for operations of the target task.

The electronic device <NUM> may further include a communication module <NUM>. A transmitter <NUM> of the communication module <NUM> may transmit sensing time points of sensing data of the plurality of sensors <NUM> and a time difference between the sensing data to the external device (e.g., the separate computing device <NUM>). A receiver <NUM> of the communication module <NUM> may receive a task result from the external device.

The timekeeping module <NUM> may calculate a task latency based on a time point when the task result is received. The task latency may include a transmission delay and a reception delay according to communication of the communication module <NUM>, and a time when the task is performed by the external device. Accordingly, the electronic device <NUM> may calculate only accurate time information including delays due to communication, and the separate computing device <NUM> may perform operations for the target task, whereby the efficiency of all computations may improve.

The operations of the electronic device <NUM> and the external device will be described in more detail with reference to <FIG>.

For example, in operations <NUM>, the first sensor <NUM> and the second sensor <NUM> may transmit sensing data to the communication module <NUM>. The communication module <NUM> may transmit the sensing data to the main processor <NUM> of the separate computing device <NUM>.

In operation <NUM>, the timekeeping module <NUM> may calculate a time difference between the sensing data based on a last timestamp of the first sensor <NUM>. In operation <NUM>, the communication module <NUM> may transmit the calculated time difference and time information including the sensing time points of the respective sensing data to the separate computing device <NUM>. In operation <NUM>, the main processor <NUM> of the separate computing device <NUM> may determine localization of the electronic device <NUM> at the last timestamp. In operation <NUM>, the main processor <NUM> may transmit the localization of an AR device (e.g., the electronic device <NUM>) to the sub-processor <NUM>.

In operation <NUM>, the sub-processor <NUM> may render an image for the determined localization. In operation <NUM>, the communication module <NUM> may receive the rendered image from the separate computing device <NUM>. In operation <NUM>, the timekeeping module <NUM> may determine a time point when the communication module <NUM> receives the rendered image to be a rendering complete time point.

In operation <NUM>, the timekeeping module <NUM> may determine a rendering latency from the timestamp corresponding to a localization time point to the time point when the rendered image is received. In operation <NUM>, the timekeeping module <NUM> may transmit the calculated rendering latency to the processor <NUM>. In operation <NUM>, the processor <NUM> may correct the rendered image using the calculated rendering latency. For example, the processor <NUM> may perform a latency compensation operation <NUM> (e.g., image warping). In operation <NUM>, the processor <NUM> may transmit a corrected image (e.g., an output image) to the display <NUM>. In operation <NUM>, the display <NUM> may output the corrected image.

<FIG> illustrates an example of correcting an image in view of a rendering latency by an electronic device for providing augmented reality (AR).

An electronic device <NUM> may be implemented as an AR glasses device as shown in <FIG>. As described above with reference to <FIG>, the electronic device <NUM> may correct a rendered image based on a rendering latency. One or more processors of the electronic device <NUM> may perform warping based on any one or any combination of any two or more of a rotation transform, a translation, and a scale change corresponding to a localization change of the electronic device <NUM> during a task latency from a sensing time point of reference sensing data to a task complete time point. The rotation transform, the translation, and the scale change may be expressed as a coordinate transformation matrix between views of the electronic device <NUM>. A view of the electronic device <NUM> may vary according to the localization (e.g., a pose and a location) of the electronic device <NUM>, and may correspond to, for example, a direction and a location viewed by a vision sensor (e.g., a camera sensor) of the electronic device <NUM>. In other words, the view of the electronic device <NUM> may be interpreted as corresponding to a camera view, and the coordinate transformation matrix between views may be expressed similarly to a coordinate transformation matrix of extrinsic camera calibration. One or more processors of the electronic device <NUM> may perform any one, any combination, or all operations described above with respect to <FIG>. Further, one or more memories of the electronic device <NUM> may store instructions that, when executed by the one or more processors, configure the one or more processors to perform any one, any combination, or all operations described above with respect to <FIG>. For example, the electronic device <NUM> may correspond to the electronic device of any one, any combination, or all of <FIG>, as a non-limiting example.

The warping may be an operation of generating an output image by applying, to the rendered image, a coordinate transformation matrix including any one or any combination of a rotation transform parameter, a translation parameter, and a scale change parameter. The coordinate transformation matrix calculated based on the rendering latency may transform coordinates according to a first coordinate system corresponding to a view (e.g., a first view direction) positioned at the localization determined at the localization timestamp into coordinates according to a second coordinate system corresponding to a view (e.g., a second view direction) positioned at localization predicted at the rendering complete time point. The electronic device <NUM> may rotate and translate coordinates of each pixel in an AR image including virtual content generated according to the first coordinate system to coordinates according to the second coordinate system described above.

For example, in <FIG>, the electronic device <NUM> may determine a localization timestamp while facing a first view direction <NUM>. Thereafter, while rendering is performed, the localization of the electronic device <NUM> may change from the first view direction <NUM> to a second view direction <NUM>. A view direction may be a direction that a reference vector of the electronic device <NUM> faces, and for example, a direction viewed by a vision sensor (e.g., a camera sensor). The electronic device <NUM> may perform warping <NUM> to transform coordinates of the virtual content generated in the first coordinate system corresponding to the first view direction <NUM> into the second coordinate system corresponding to the second view direction <NUM>. Accordingly, the electronic device <NUM> may output virtual content <NUM> at world coordinates that accurately fit a real object <NUM> (e.g., a desk), despite a localization change <NUM> during the rendering latency.

<FIG> illustrates a method of independently managing and using a sensing time point for localization.

First, in operation <NUM>, an electronic device may generate sensing data for localization through sensing by a plurality of sensors. For example, the electronic device may generate the reference sensing data by performing one of an operation of capturing a camera image by a camera sensor, an operation of sensing radar data by a radar sensor, and an operation of sensing a lidar image by a lidar sensor. The electronic device may calibrate a sensing time point of the camera sensor based on either one or both of an exposure time and a read time of the camera image. In addition, the electronic device may generate other sensing data by performing any one or any combination of any two or more of an operation of sensing an inertia of the electronic device by an IMU, an operation of performing sensing by an active sensor in response to receiving a trigger input, and an operation of performing sensing by a passive sensor at predetermined intervals.

In operation <NUM>, the electronic device may determine a sensing time point of reference sensing data generated through sensing by a reference sensor and a sensing time point of other sensing data of another sensor, among the plurality of sensors, based on a clock rate of a timekeeping module.

In operation <NUM>, the electronic device may calculate a time difference between the sensing time point of the reference sensing data and the sensing time point of the other sensing data.

In operation <NUM>, the electronic device may determine a task latency from the sensing time point of the reference sensing data to a task complete time point based on the clock rate of the timekeeping module. For example, the electronic device may determine a rendering latency from the sensing time point of the reference sensing data to a time point when rendering of an image for AR is completed.

In operation <NUM>, the electronic device may correct, based on the task latency, a task result processed according to localization of the electronic device determined for the sensing time point of the reference sensing data based on the calculated time difference, the reference sensing data, and the other sensing data. For example, the electronic device may generate an output image by correcting an image rendered according to localization of the electronic device determined for the sensing time point of the reference sensing data based on the rendering latency. As described with reference to <FIG>, the electronic device may perform warping based on any one or any combination of any two or more of a rotation transform, a translation, and a scale change corresponding to a localization change of the electronic device during a task latency from a sensing time point of reference sensing data to a task complete time point.

For example, the electronic device may determine the localization of the electronic device for the sensing time point of the reference sensing data based on the calculated time difference, the reference sensing data, and the other sensing data. The electronic device may generate a rendered image as a task result for the localization.

As another example, the electronic device may request rendering from an external device (e.g., a separate computing device) and receive the rendered image from the external device. In this case, the electronic device may transmit the sensing time points of the sensing data of the plurality of sensors and the time difference to the external device, in operation <NUM> described above. Further, the electronic device may receive the task result from the external device and calculate the task latency based on a time point when the task result is received, in operation <NUM>.

In operation <NUM>, the electronic device may output the corrected task result based on the task latency. For example, the electronic device may display an output image on a display.

The operation of the electronic device is not limited to the above description and may be performed together with the operations described above with reference to <FIG> in parallel or in a time-series manner.

The electronic device may be applied to any type of device in which a sensor system requiring time synchronization is mounted. The timekeeping module may be implemented as hardware in the form of a chip or may be a combination of hardware and software. In an example, the electronic device may be implemented as a wireless communication device equipped with a camera and an IMU, a sensor system for vehicles, and an AR device. Furthermore, the electronic device may be applied to robot control systems, and all electronic systems machine/aviation/maritime control.

The electronic device of one or more embodiments may measure and record an accurate time of data measured by each sensor and an accurate generation time of an operation result using the measured data. The electronic device of one or more embodiments may correct an error in motion estimation using the measured time and improve a delay of a rendering result. For the accurate calculation of the estimated delay, the electronic device of one or more embodiments may store and calibrate pre-calculated measurement time errors (e.g., calibration information) according to the characteristics of the sensors.

The electronic devices, sensors, timekeeping modules, processors, output modules, first sensors, second sensors, memories, main processors, sub-processors, displays, timekeeping processors, communication modules, transmitters, receivers, separate computing devices, camera sensors, IMUs, passive sensors, active sensors, electronic device <NUM>, sensor <NUM>, timekeeping module <NUM>, processor <NUM>, output module <NUM>, memory <NUM>, electronic device <NUM>, sensors <NUM>, first sensor <NUM>, second sensor <NUM>, timekeeping module <NUM>, processor <NUM>, memory <NUM>, processor <NUM>, main processor <NUM>, sub-processor <NUM>, display <NUM>, sensors <NUM>, camera sensor <NUM>, IMU <NUM>, passive sensor <NUM>, active sensor <NUM>, timekeeping module <NUM>, main processor <NUM>, sub-processor <NUM>, display <NUM>, electronic device <NUM>, sensors <NUM>, first sensor <NUM>, second sensor <NUM>, timekeeping module <NUM>, timekeeping processor <NUM>, memory <NUM>, processor <NUM>, display <NUM>, communication module <NUM>, transmitter <NUM>, receiver <NUM>, separate computing device <NUM>, main processor <NUM>, sub-processor <NUM>, electronic device <NUM>, and other apparatuses, units, modules, devices, and components described herein with respect to <FIG> are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a fieldprogrammable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
An electronic device, comprising:
a timekeeping processor (<NUM>, <NUM>) configured to:
determine a sensing time point of reference sensing data, generated through sensing by a reference sensor (<NUM>, <NUM>) of a plurality of sensors (<NUM>, <NUM>), and a sensing time point of other sensing data of another sensor (<NUM>, <NUM>) of the plurality of sensors (<NUM>, <NUM>), based on a clock rate of the timekeeping processor (<NUM>, <NUM>);
determine a time difference between the sensing time point of the reference sensing data and the sensing time point of the other sensing data; and
determine, based on the clock rate of the timekeeping processor, a task latency from the sensing time point of the reference sensing data to a task complete time point of a task result processed according to a localization of the electronic device determined for the sensing time point of the reference sensing data based on the determined time difference, the reference sensing data, and the other sensing data; and
one or more other processors (<NUM>, <NUM>) comprising a main processor and a sub-processor;
wherein the main processor (<NUM>) is configured to generate the localization of the electronic device for the sensing time point of the reference sensing data based on the determined time difference, the reference sensing data, and the other sensing data, and
the sub-processor (<NUM>) is configured to generate a rendered image as a task result for the localization;
wherein the main processor (<NUM>) is further configured to predict a localization change of the electronic device based on the task latency, and to perform warping of the rendered image corresponding to the predicted localization change in order to correct, based on the task latency, the task result.