Patent ID: 12198362

Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order. Also, descriptions of features that are known after an understanding of the disclosure of this application may be omitted for increased clarity and conciseness.

The features described herein may be embodied in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.

The terminology used herein is for describing various examples only and is not to be used to limit the disclosure. The articles “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises,” “includes,” and “has” specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements, and/or combinations thereof.

Throughout the specification, when a component is described as being “connected to,” or “coupled to” another component, it may be directly “connected to,” or “coupled to” the other component, or there may be one or more other components intervening therebetween. In contrast, when an element is described as being “directly connected to,” or “directly coupled to” another element, there can be no other elements intervening therebetween. As used herein, the term “and/or” includes any one and any combination of any two or more of the associated listed items.

Although terms such as “first,” “second,” and “third” may be used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms. Rather, these terms are only used to distinguish one member, component, region, layer, or section from another member, component, region, layer, or section. Thus, a first member, component, region, layer, or section referred to in the examples described herein may also be referred to as a second member, component, region, layer, or section without departing from the teachings of the examples.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein. The use of the term “may” herein with respect to an example or embodiment (e.g., as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto

FIG.1illustrates an example operation of an electronic apparatus with image processing, according to one or more embodiments. Referring toFIG.1, an electronic apparatus100may generate a depth image102based on input images101.

The input images101may be captured by a time-of-flight (TOF) sensor, e.g., with the electronic apparatus being representative of including the TOF sensor, as well as one or more processors configured to perform image processing dependent on the input images101and to generate the depth image102. The input images101may be respective images of captured infrared rays of different phases. For example, a first input image101may be captured by the TOF sensor based on receipt of first infrared rays with a phase modulation of 0 degrees)(°, a second input image101may be captured by the TOF sensor based on receipt of second infrared rays with a phase modulation of 90°, a third input image101may be captured by the TOF sensor based on receipt of third infrared rays with a phase modulation of 180°, and a fourth input image101may be captured by the TOF sensor based on receipt of fourth infrared rays with a phase modulation of 270°, as non-limiting examples. Thus, each of the input images101may be classified based on their respective reference phases of their respective infrared rays. For example, the first input image101obtained by the example phase modulation of 0° may be classified as Q1, the second input image101obtained by the example phase modulation of 90° may be classified as Q2, the third input image101obtained by the example phase modulation of 180° may be classified as Q3, and the fourth input image obtained by the example phase modulation of 270° may be classified as Q4.

The electronic apparatus100may determine a real part image, an imaginary part image, and a phase image, for example, based on the input images101and the electronic apparatus100may generate the depth image102based on the phase image. An indirect TOF (iTOF) method for measuring depth information using a phase difference may obtain depth information of high resolution at low cost. However, the accuracy may be lowered or less than optimum in such a process of calculating a phase difference using a plurality of images. Rather, electronic apparatus100may obtain reliable depth information, e.g., without such a lowering of the accuracy, by removing noise from an image associated with the depth image102.

FIG.2illustrates an example real part image, an example imaginary part image, and an example phase image, according to one or more embodiments. Referring toFIG.2, a phase image203may be generated based on a real part image201and an imaginary part image202, for example. The real part image201, the imaginary part image202, and the phase image203may be determined by Equations 1 through 3 below, for example.

α=12⁢(Q1-Q3)Equation⁢1β=12⁢(Q2-Q4)Equation⁢2ϕ=arctan⁢βαEquation⁢3

Here, a denotes the real part image201, β denotes the imaginary part image202, and ϕ denotes the phase image203. Q1through Q4denote input images based on infrared rays of the example different phases. For example, Q1may correspond to 0°, Q2may correspond to 90°, Q3may correspond to 180°, and Q4may correspond to 270° when using phases of 0°, 90°, 180°, and 270° as the non-limiting example different phases of the respective captured infrared rays, e.g., by the aforementioned TOF sensor. For ease of explanation, the input images Q1and Q3may be considered a first image pair, and the input images Q2and Q4may be considered a second image pair. In this example, the real part image201may be determined based on a difference in the first image pair, and the imaginary part image202may be determined based on a difference in the second image pair.

Phase wrapping, in which phase information is reset for every preset period (e.g., 2π), may occur in the phase image203, and thus a streak of the phase image203may occur. Although a statistical noise characteristic may be analyzed to effectively remove most noise, the streak may not be suitable or may not be desirable for analyzing a distribution characteristic.

FIG.3illustrates an example amplitude image and an example offset image, according to one or more embodiments. Referring toFIG.3, an amplitude image301and an offset image302may be generated based on a real part image and an imaginary part image. Example equations 4 and 5 below may be used to generate the amplitude image301and the offset image302.

A=α2+β2Equation⁢4I=14⁢(Q1+Q2+Q3+Q4)Equation⁢5

Here, A denotes the amplitude image301, and I denotes the offset image302. The offset image302may correspond to an average of input images, for example.

The real part image, the imaginary part image, the amplitude image301, and the offset image302may each include degradation by noise. However, the amplitude image301and the offset image302may each have a higher signal-to-noise ratio (SNR), e.g., compared to respective SNRs of the real part image and the imaginary part image, because the amplitude image301and the offset image302may respectively be dependent on the real part image and the imaginary part image, e.g., dependent on a sum of the real part image and the imaginary part image. Thus, an analyzed noise characteristic derived from the amplitude image301and/or the offset image302may be effectively applied for removing noise from the real part image and/or the imaginary part image.

For example, a graph310ofFIG.3may represent a ratio of pixel values of the amplitude image301and the offset image302. A vertical axis may correspond to a pixel value of the amplitude image301and may be referred to as the amplitude axis, and a horizontal axis may correspond to a pixel value of the offset image302and may be referred to as the DC axis. Referring to the graph310, a pixel value distribution of the amplitude image301compared to the offset image302is mostly matched in the form of y=x as shown in a region311, which indicates a high correlation between the amplitude image301and the offset image302.

However, region312of the graph310may represent portions of the amplitude image301that may have a pixel value lower than corresponding pixels of the offset image302. As a distance between a sensor and an object increases, a pixel value corresponding to the object may decrease, and a corresponding pixel value may further decrease when calculating the real part image and the imaginary part image. Thus, a pixel value corresponding to the amplitude image301may also decrease. The offset image302may be based on a sum of all input images, and a signal thereof may have more energy and a greater pixel value than other images including, for example, the amplitude image301which may be dependent on the summation of the squares of the real part image and the imaginary part image. That is, the offset image302may have a higher SNR than each of the real part image, the imaginary part image, and the amplitude image301. In an example, the electronic apparatus may remove noise using the offset image302as a guide, thereby improving noise removal performance.

FIG.4illustrates an example of a distribution characteristic of offset images, according to one or more embodiments. Referring toFIG.4, an image401may represent an average (e.g., temporal average) of sample offset images, and an image402may represent a variance (e.g., temporal variance) of the sample offset images. In a state in which there is no movement, a plurality of sample offset images (e.g., 400 images) having the same pixel values of all amplitudes may be captured. The image401may be determined based on the determined average of pixels of the sample offset images, and the image402may be determined based on a determined variance of the pixels.

For example a graph410ofFIG.4may represent a relationship between the average and the variance of the sample offset images. A noise standard deviation may be greater in an area having a greater amplitude when using the sample offset images having depths and amplitudes that are uniformly distributed without a strong edge, for example. In this example, the graph410may represent an amplitude average distribution and a noise variance distribution. The graph410may exhibit characteristics of a Poisson distribution in which a noise variance is proportional to an amplitude value. This may be expressed by Equation 6 below, for example.
I=Ī+η(Ī)δ  Equation 6:

Here, I denotes a sample offset image with noise, Ī denotes a sample offset image with the noise removed, η(Ī) denotes a standard deviation of the noise, and δ denotes a Gaussian noise with an average of 0 and a standard deviation of 1. When using a plurality of sample offset images (e.g., 400 images), the electronic apparatus may estimate that an average image has almost no noise. Thus, Ī may correspond to the average image.

The noise variance may be proportional to an amplitude value as expressed by Equation 7 below, for example.
η2(Ī)=aĪEquation 7:

In Equation 7, a denotes a slope.

In an example, an electronic apparatus may perform noise removal using a non-local mean (NLM) filter. As a non-limiting example, a structure of the NLM filter may be expressed by Equations 8 and 9 below.

I^i=∑j∈𝒩i1Wi⁢wij⁢IjEquation⁢8Wi=∑j∈𝒩iwijEquation⁢9

Here, Ī denotes an offset image from which noise is removed, Nidenotes a neighborhood window centered at i, and i, j denotes a pixel position. I denotes a noisy input offset image (e.g., an initial offset image), wijdenotes a weighted sum coefficient of a position j in a central window of a position i, and Widenotes a normalization coefficient.

A noise removal performance by the electronic apparatus may depend on a design of the weighted sum coefficient wij. For example, a noise characteristic of an offset image may be defined based on a Poisson noise characteristic, and such a noise characteristic of the offset image may be reflected in the design of wij. A stochastic distance corresponding to a Poisson distribution may be used to calculate a distance between NLM patches, and a relationship between an amplitude value and a noise intensity analyzed as represented by the graph410may be applied to a noise removal intensity. Thus, the noise removal performance by the electronic apparatus may be maximized.

In an example, the weighted sum coefficient wijmay be defined as expressed by Equation 10 below, for example.

wi⁢j=exp⁡(-∑k∈χdψ(Ii+k,Ij+k)γ·σηi)Equation⁢10

In Equation 10, a stochastic distance dip may correspond to a Kullback-Leibler distribution and may be expressed by Equation 11. χ denotes an aggregate position of pixels in a surrounding area of i, j coordinates, σηidenotes a noise standard deviation, and γ denotes a value that may be adjusted with a noise removal constant.

dψ(Ii,Ij)=12[(Ii-Ij)·ln⁡(IiIj)]Equation⁢11

Here, dψ may correct a distance value in a Poisson noise by adding a ratio component to a difference between the two values. σηimay be expressed by Equation 12 below.
σηi=a·{circumflex over (λ)}ML(Ii)  Equation 12:

Here, {circumflex over (λ)}MLmay be expressed by the example Equation 13 below, and a may represent a slope.

λˆM⁢L(Ii)=1n⁢∑k∈χIi+kEquation⁢13

FIG.5illustrates an example of noise removal-based image processing, according to one or more embodiments. Referring toFIG.5, in operation510, an electronic apparatus may receive input images that are based on infrared images of different phases. The input images may be generated by a TOF sensor and transmitted to a processor of the electronic apparatus. The input images may be generated through infrared rays of reference phases including 0°, 90°, 180°, and 270°, as non-limiting examples.

In operation520, the electronic apparatus may determine a real part image, an imaginary part image, and an offset image based on the input images. The electronic apparatus may determine the real part image based on a difference in a first image pair among the input images, the imaginary part image based on a difference in a second image pair among the input images, and the offset image based on a sum of the input images, for example.

In operation530, the electronic apparatus may remove noise from each of the real part image and the imaginary part image using the offset image as a guide for the noise in the real part image and the imaginary part image. In an example, the electronic apparatus may perform such a noise removal using an NLM filter. The electronic apparatus may determine a weighted sum coefficient of the NLM filter from the offset image and calculate a weighted sum for each of the real part image and the imaginary part image based on the weighted sum coefficient.

The electronic apparatus may define a target patch and a scan patch of a preset size in the offset image and determine a weight of a central pixel of the scan patch based on a determined similarity between the target patch and the scan patch. The electronic apparatus may determine a weighted sum coefficient of the center pixel of the target patch based on the weight of the center pixel of the scan patch associated with the target patch.

The electronic apparatus may determine the weight by changing the target patch and the scan patch. For example, the electronic apparatus may determine scan patches for a target patch while sliding a window in/across the offset image, and calculate a similarity between the target patch and each of the scan patches. The electronic apparatus may determine a weighted sum coefficient of a central pixel of the target patch based on each calculated similarity. The electronic apparatus may calculate a similarity and a weighted sum coefficient for remaining target patches in a similar manner as described above.

The electronic apparatus may apply a weighted sum coefficient derived from a guide image (e.g., offset image) to a filter input image (e.g., real part image and imaginary part image). The electronic apparatus may remove noise of a target image by calculating a weighted average between patches based on a weight of each pixel of the filter input image.

In an example, the electronic apparatus may remove noise from the filter input image through twicing regularization. Twicing regularization may be a method of obtaining a final filtering result by obtaining a residual image corresponding to a difference between a temporary filtering result of the filter input image and the filter input image and adding, to the temporary filtering result, a result of additionally applying a filter of the same coefficient to the residual image. Twicing regularization may be expressed by Equation 14 below, for example.
+{circumflex over (I)}+GNR(I−Î,I),{circumflex over (I)}=GNR(I,I)  Equation 14:

Here,denotes a final filtering result, Î denotes a temporary filtering result, and I denotes a filter input image. Equation 14 may represent twicing regularization in a case in which both the filter input image and the guide image are offset images. However, it may also be applied when the filter input image is a real part image or an imaginary part image. A grain-noise reduction (GNR) (T, G) filter may refer to a filter that removes noise from a filter input image T using a guide image G. Twicing normalization may be used to achieve both purposes of noise removal and definition maintenance by supplementing a high frequency component that may be lost when noise removal is performed by the electronic apparatus, using a residual image.

For example, the electronic apparatus may generate a temporary filtering result by removing noise from a real part image using an offset image as a guide, and determine a residual image corresponding to a difference between the temporary filtering result and the real part image. The electronic apparatus may generate an improved residual image by removing noise from the residual image using the offset image as the guide and determine an improved real part image by adding the temporary filtering result and the improved residual image. The electronic apparatus may determine an improved imaginary part image by performing twicing regularization based on the offset image and the imaginary part image.

In an example, the electronic apparatus may perform recursive twicing regularization. The recursive twicing regularization may include a plurality of noise removal stages in which noise removal of the real part image and/or the imaginary part image is performed based on an offset image. Operations of a block501may correspond to such recursive twicing regularization.

In the recursive twicing regularization, a filter input image may include a real part image, an imaginary part image, and an offset image, and these images may be repeatedly improved by respective noise removals until a final stage of noise removal is completed. Thus, noise removal may be gradually performed on the offset image through the noise removal stages, and different versions of the improved offset image may be used at the stages.

The recursive twicing regularization may be expressed by Equation 15 below, for example.

=+GNR⁡(-,),=GNR⁡(,)Equation⁢15

Here,denotes a final filtering result at a Kth stage,denotes a temporary filtering result at the Kth stage, anddenotes a final filtering result at a K−1th stage.may correspond to a filter input image at the Kth stage.

Equation 15 may represent recursive twicing regularization in a case in which both the filter input image and the guide image are offset images. However, it may also be applied when the filter input image is the guide image, the real part image, and the imaginary part image. This may be expressed by the below example Equations 16 through 21.

Equations 16 and 17 below may represent example recursive twicing regularization for an offset image.
=+GNR(Rk-1I,)  Equation 16:

In Equation 16, Rk-1Idenotes a residual image corresponding to a difference between a temporary filtering resultat a Kth stage and a final filtering resultat a K−1th stage. Rk-1Imay be expressed by Equation 17 below, for example.
Rk-1I=−GNR(,)  Equation 17:

According to Equation 17, a temporary filtering resultmay be determined by removing noise from a filter input imageusing an offset imageas a guide, and a residual image Rk-1Icorresponding to a difference in the temporary filtering result, and the filter input imagemay be determined. According to Equation 16, an improved residual image may be determined by removing noise from the residual image Rk-1Iusing the offset imageas the guide, and the final filtering resultmay be determined through a sum of the temporary filtering resultand the improved residual image.

Equations 18 and 19 below may represent example recursive twicing regularization for a real part image.
=+GNR(Rk-1α,)  Equation 18:
Rk-1α=−GNR(,)  Equation 19:

Here,denotes a final filtering result at a Kth stage,denotes a temporary filtering result at the Kth stage, anddenotes a final filtering result at a K−1th stage.may correspond to the filter input image at the Kth stage.

According to Equation 19, a temporary filtering resultmay be determined by removing noise from a filter input imageusing an offset imageas a guide, and a residual image Rk-1αcorresponding to a difference between the temporary filtering resultand the filter input imagemay be determined. According to Equation 18, an improved residual image may be determined by removing noise from the residual image Rk-1Iusing the offset imageas the guide, and the final filtering resultmay be determined through a sum of the temporary filtering resultand the improved residual image.

Equations 20 and 21 below may represent example recursive twicing regularization for an imaginary part image.
=+GNR(Rk-1β,)  Equation 20:
Rk-1β=−GNR(,)  Equation 21:

The electronic apparatus may perform noise removal based on Equations 18 through 21 in operation530ofFIG.5and determine whether a current stage is a final stage in operation540. For example, the final stage may be set based on the number of stages (or the number of repetitions), a threshold associated with a difference before and after the noise removal, and the like. When the current stage is not the final stage, the electronic apparatus may update the offset image based on Equations 16 and 17 in operation550and perform noise removal based on the updated offset image in operation530.

For example, the electronic apparatus may determine a first improved real part image, a first improved imaginary part image, and a first improved offset imageby performing twicing regularization on the real part image, the imaginary part image, and the offset image, respectively, based on the offset imagein a Kth noise removal stage. The electronic apparatus may also determine a second improved real part image, a second improved imaginary part image, and a second improved offset imageby performing twicing regularization on the first improved real part image, the first improved imaginary part image, and the first improved offset image, respectively, based on the offset imagein a K+1th noise removal stage.

As the stages of noise removal progress, noise of the real part image and the imaginary part image may be repeatedly removed, and an offset image repeatedly updated in such noise removal stages may be used.

When the current stage is the final stage, the electronic apparatus may generate a depth image based on the improved real part image and the improved imaginary part image. When a final improved real part image and a final improved imaginary part image are determined at the final noise removal stage (e.g., determined in operation540), the electronic apparatus may generate a depth image based on the improved final real part image and the improved final imaginary part image. The electronic apparatus may determine a phase image based on the improved real part image and the improved imaginary part image and generate the depth image by performing phase wrapping based on the phase image.

The depth image may be generated by Equations 22 through 25 below, for example.

α^i=∑j∈𝒩i1Wi⁢wij⁢αjEquation⁢22β^i=∑j∈𝒩i1Wi⁢wij⁢βjEquation⁢23ϕ^=arctan⁢β^α^Equation⁢24D^=c2⁢f·ϕ^2⁢πEquation⁢25

Here, {circumflex over (α)}, {circumflex over (β)}, and {circumflex over (ϕ)} may be a real part image, an imaginary part image, and a phase image, respectively, from which noise has been removed, that is, they may be the final improved real part image, the final improved imaginary part image, and the final improved phase image. {circumflex over (D)} may be a depth image from which noise is removed, e.g., in which noise has not been introduced, and may be referred to as an improved depth image. f denotes a frequency of an infrared ray used as a light source.

FIG.6illustrates an example offset image and examples of improved offset images, according to one or more embodiments. An offset image601may be an initial unimproved offset image. An improved offset image602may be a version of an offset image improved without recursive twicing regularization, and an improved offset image603may be a version of an offset image improved through recursive twicing regularization. The improved offset image603may include less noise and may include sharp edges, and have objects shown vividly despite the objects being disposed a long distance from the sensor, e.g., the TOF sensor of the electronic apparatus. While discussions above have been presented with respect to the use of the offset image as the guide, such improvements of the real part image, imaginary part image, and phase image, as well as twice regularization(s), are also available using the amplitude image as the guide for noise removal,

FIG.7illustrates examples of different filtering results, according to various embodiments. Referring toFIG.7, there are an original depth image701that could be generated by original real and imaginary part images in which noise removal has not been performed, while depth images702and704are depth images respectively generated based on amplitude image-based noise removal having been applied according to one or more embodiments, e.g., where the above noise removal is performed using the amplitude image as the guide (in place of the offset image), and while depth images703and705are depth images respectively generated based on offset image-based noise removal having been applied according to one or more embodiments, e.g., where the above noise removal is performed using the offset image as the guide. The depth images702and703may be depth images resulting from the recursive twicing regularization having not been applied, while the depth images704and705may be depth images resulting from such application of the recursive twicing regularization. For example, the depth image705may provide relatively accurate depth information with high resolution.

FIG.8illustrates an example electronic apparatus with image processing, according to one or more embodiments. Referring toFIG.8, an electronic apparatus800includes a processor810and a memory820. The memory820may be connected to the processor810and store instructions executable by the processor810, store data to be processed by the processor810, and/or store data processed by the processor810. The memory820may include a non-transitory computer-readable medium, for example, a high-speed random-access memory (RAM) and/or nonvolatile computer-readable medium (e.g., at least one disk storage device, flash memory device, or another nonvolatile solid-state memory device). The electronic apparatus800may correspond to any or all electronic apparatuses described herein. The electronic apparatus800may also be an apparatus that can perform multiple functions in addition to such depth image generation with such lessened noise, and may alternatively be a hardware image processing module of such an electronic apparatus.

The processor810may execute instructions stored in the memory820, for example, which when executed may configure the processor810to perform any one or any combination of two or more or all operations described herein with reference toFIGS.1through10. For example, the processor810may receive input images based on infrared rays having different phases, determine a real part image, an imaginary part image, and an offset image based on the input images, remove noise from each of the real part image and the imaginary part image using the offset image as a guide, and generate a depth image based on an improved real part image and an improved imaginary part image corresponding to the result of the noise removal. For further detailed descriptions of the electronic apparatus800, reference may be made to what is additionally described herein with reference toFIGS.1through7and9-10, which are also applicable to the electronic apparatus800.

FIGS.9and10illustrate examples of an electronic apparatus, respectively according to one or more embodiments. Referring toFIG.9, the electronic apparatus900includes a TOF sensor910and an image processing apparatus920. The image processing apparatus920is representative of a processor and a memory. The TOF sensor910may be representative of including an infrared light source configured to transmit infrared rays of different phases and a sensor configured to respectively capture infrared images with the different phases. An infrared image may be provided to the image processing apparatus920as an input image, and the image processing apparatus920may generate a depth image based on the input image. For example, the processor of the image processing apparatus920may determine a real part image, an imaginary part image, and an offset image based on the input images, remove noise from each of the real part image and the imaginary part image using the offset image as a guide, and generate a depth image based on an improved real part image and an improved imaginary part image corresponding to a result of the removing. For further detailed descriptions of the electronic apparatus900, reference may be made to what is additionally described herein with reference toFIGS.1through8and10, which are also applicable to the electronic apparatus900. The electronic apparatus900may correspond to any or all electronic apparatuses described herein. The electronic apparatus800may also be an apparatus that can perform multiple functions in addition to such depth image generation with such lessened noise, and may alternatively be a hardware TOF module of such an electronic apparatus, where the TOF module performs the transmission of the infrared rays with different frequencies, collects received reflections from the transmitted infrared rays, and generates the depth image with the lessened noise due to any one or more or all noise removal operations descried herein.

Referring toFIG.10, an electronic apparatus1000includes a processor1010, a memory1020, a camera1030, a storage device1040, an input device1050, an output device1060, and a network interface1070, and these components may communicate with one another through a communication bus1080, as non-limiting examples. As a non-limiting example, the electronic apparatus1000may be, or may be implemented as at least a portion of, a mobile device such as a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, and the like, a wearable device such as a smart watch, a smart band, smart glasses, and the like, a home appliance such as a television (TV), a smart TV, a refrigerator, and the like, a security device such as a door lock and the like, and a vehicle such as an autonomous vehicle, a smart vehicle, and the like. The electronic apparatus1000may be any one or more or all of the electronic apparatuses described herein, e.g., the electronic apparatus1000may be the electronic apparatus100ofFIG.1, the electronic apparatus800ofFIG.8, and/or the electronic apparatus900ofFIG.9, as non-limiting examples.

The processor1010may execute instructions to control operations of the electronic apparatus1000, as a non-limiting example. The processor1010may be configured to perform any one, any combination of two or more or all of the operations described herein with reference toFIGS.1through10. For example, the processor1010may execute such instructions stored in the memory1020and/or the storage device1040, which when executed by the processor1010, configure the processor to perform any one, any combination of two or more or all of the operations described herein with reference toFIGS.1through10. In an example, the memory1020may include a computer-readable storage medium or device. For example, the memory1020may store instructions executed by the processor1010, and may further store related information while other software and/or applications are being executed by the electronic apparatus1000. For example, the processor1010may execute such further instructions which control other operations of the electronic apparatus1000, e.g., in addition to the image processing and/or the depth calculation operations, such as further operations that utilize the calculated depths and/or calculated depth image(s) and other operations of the electronic apparatus that do not involve the calculated depths and/or calculated depth image(s).

The camera1030may capture an image and/or video, and is representative of one or more cameras1030. As a non-limiting example, the camera1030may include a TOF camera. The storage device1040may include a computer-readable storage medium or device. The storage device1040may store a greater amount of information than the memory1020and may store the information for a long period of time. As non-limiting examples, the storage device1040may include, for example, a magnetic hard disk, an optical disc, a flash memory, a floppy disk, or other types of nonvolatile memory.

As a non-limiting example, the input device1050may receive an input from a user using a keyboard and/or a mouse, as well as through use of a touch input, a voice input, and an image input, as non-limiting examples. The input device1050may include, for example, a keyboard, a mouse, a touch screen, a microphone, or any other device that detects an input from a user to the electronic apparatus1000, as non-limiting examples. The output device1060may provide an output of the electronic apparatus1000, e.g., to a user, through a visual, auditory, and/or tactile channel, or through other transmission channel. As non-limiting examples of the visual, auditory, and/or tactile channels, the output device1060may include, for example, a display, a touch screen, a speaker, a vibration generating device, or any other device that provides an output to a user. The network interface1070may communicate with an external device through a wired and/or wireless network.

The image processing apparatuses, the electronic apparatuses, image processing modules, devices, or apparatuses, TOF sensors, TOF modules, processors, memories, storage devices, input devices, output devices, network interfaces, communication busses, and other apparatuses, devices, units, modules, and components described herein with respect toFIGS.1through10are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term “processor” or “computer” may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated inFIGS.1-10that perform the operations described in this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

Instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler. In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computer using an interpreter. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions herein, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.

The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computers.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents.

Therefore, in addition to the above disclosure, the scope of the disclosure may also be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.