Patent ID: 12243196

The same reference numerals are used to represent the same elements throughout the drawings.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of various embodiments of the disclosure as defined by the claims and their equivalents. It includes various specific details to assist in that understanding, but these are to be regarded as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the various embodiments described herein can be made without departing from the scope and spirit of the disclosure. In addition, descriptions of well-known functions and constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the disclosure. Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of the disclosure is provided for illustration purposes only and not for the purpose of limiting the disclosure as defined by the appended claims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to “a component surface” includes reference to one or more of such surfaces.

The embodiments described herein are not necessarily mutually exclusive, as some embodiments of the disclosure can be combined with one or more embodiments of the disclosure to form new embodiments. The term “or” as used herein, refers to a non-exclusive or, unless otherwise indicated. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein can be practiced and to further enable those skilled in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

As is traditional in the field, embodiments may be described and illustrated in terms of blocks which carry out a described function or functions. These blocks, which may be referred to herein as units or modules or the like, are physically implemented by analog or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like. The circuits constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block. Each block of the embodiments may be physically separated into two or more interacting and discrete blocks without departing from the scope of the disclosure. Likewise, the blocks of the embodiments may be physically combined into more complex blocks without departing from the scope of the disclosure.

The accompanying drawings are used to help easily understand various technical features and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. As such, the disclosure should be construed to extend to any alterations, equivalents and substitutes in addition to those which are particularly set out in the accompanying drawings. Although the terms first, second, or the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are generally only used to distinguish one element from another.

Accordingly, embodiments herein disclose a method for producing a media file with a blur effect. The method includes receiving, by an electronic device, an image frame when the electronic device captures at least one image. Further, the method includes segmenting, by the electronic device, the image frame into a plurality of segments. Further, the method includes determining, by the electronic device, a segment from the plurality of segments including a foreground region of a ROI and/or a background region of the ROI. Further, the method includes detecting, by the electronic device, whether the foreground region of the ROI and/or the background region of the ROI includes motion information and static information. Further, the method includes automatically applying, by the electronic device, a motion type blur effect on the foreground region of the ROI and/or the background region of the ROI includes the motion information and a static type blur effect on the foreground region of the ROI and/or the background region of the ROI includes the static information. Further, the method includes generating, by the electronic device, the media file based on the applied the motion type blur effect and the static type blur effect. Further, the method includes storing, by the electronic device, the media file.

Embodiments herein disclose the electronic device for producing the media file with the blur effect. The electronic device includes a media effect controller coupled with a processor and a memory. The media effect controller is configured to receive the image frame. Further, the media effect controller is configured to segment the image frame into the plurality of segments. Further, the media effect controller is configured to determine the segment from the plurality of segments including the foreground region of the ROI and/or the background region of the ROI. Further, the media effect controller is configured to detect whether the foreground region of the ROI and/or the background region of the ROI includes the motion information and the static information. Further, the media effect controller is configured to automatically apply the motion type blur effect on the foreground region of the ROI and/or the background region of the ROI includes the motion information and the static type blur effect on the foreground region of the ROI and/or the background region of the ROI includes the static information. Further, the media effect controller is configured to generate the media file based on the applied the motion type blur effect and the static type blur effect. Further, the media effect controller is configured to store the media file.

The electronic device, systems, and the proposed method allows the electronic device to generate the media file by automatically applying the motion type blur effect on the foreground region of the ROI and/or the background region of the ROI that contains the motion information and the static type blur effect on the foreground region of the ROI and/or the background region of the ROI that contains the static information. As a result, a multi-type blurring effect is created in real-time in the media file, enhancing the user's experience.

The electronic devices, systems, and the proposed method allows the electronic device to optimize a color spectrum region for each segment of the plurality of segments by expanding a wavelength range of generated color spectrum region for each segment to enhance a value of each pixel of each segment. The plurality of segments includes the foreground region of the ROI and/or the background region of the ROI. The foreground region of the ROI and the background region of the ROI of the image frame is obtained by using a one-of-a manual selection of a user of the electronic device and/or by applying the AI model on the image frame.

Referring now to the drawings, and more particularly toFIGS.2to4,5A to5C,6A to6D,7A,7B,7C,7D,8A,8B,9A, and9B, where similar reference numbers denote corresponding features consistently throughout the figures, there are shown embodiments of the disclosure.

FIG.2illustrates a block diagram of an electronic device for producing a media file with the Bokeh effect, according to an embodiment of the disclosure.

Referring toFIG.2, the electronic device100can be, for example, but not limited to a smartphone, a laptop, an internet of things (IoT) device, a drone, an action camera, a sports camera, or alike.

In an embodiment, the electronic device100may include a memory110, a processor120, a communicator130, a display140, a camera150, and a media effect controller160.

The memory110stores the media file, data corresponding to a color spectrum region for each segment of an image frame, data corresponding to a wavelength range of the generated color spectrum region to enhance a value of each pixel of each segment, a wavelength value for each pixel of each segment, data corresponding to a wavelength range difference for each color for each segment, data corresponding to a typical wavelength range and data corresponding to a typical wavelength range difference for each color for each segment, data corresponding to an absolute difference between determined wavelength range difference and the typical wavelength range difference for each color for each segment, a range change ratio for each color for each segment, a first-new RGB value for each pixel, a second-new RGB value for each pixel, motion information and static information of each segment, a hash value, and a minimum change in color or a maximum change ratio associated with calculated absolute difference.

The memory110also stores instructions to be executed by the processor120. The memory110may include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard disks, optical discs, floppy disks, flash memories, or forms of erasable programmable read only memories (EPROMs) or electrically erasable and programmable ROMs (EEPROM). In addition, the memory110may, in some examples, be considered a non-transitory storage medium. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted that the memory110is non-movable. In some examples, the memory110can be configured to store larger amounts of information. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in Random Access Memory (RAM) or cache). The memory110can be an internal storage unit or it can be an external storage unit of the electronic device100, a cloud storage, or any other type of external storage.

The processor120communicates with the memory110, the communicator130, the display140, the camera150, and the media effect controller160. The processor120is configured to execute instructions stored in the memory110and to perform various processes. The processor may include one or a plurality of processors, maybe a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or an AI dedicated processor such as a neural processing unit (NPU).

The communicator130includes an electronic circuit specific to a standard that enables wired or wireless communication. The communicator130is configured for communicating internally between internal hardware components and with external devices via one or more networks. The display140is configured to display the generated media file on a screen of the electronic device100. The camera150includes a primary camera150aand a secondary camera150b-150nto capture an image of an image frame.

The media effect controller160may be implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may be embodied in one or more semiconductors. In an embodiment, the media effect controller160may be combined to the processor120and the processor120may perform all the functions conducted by the media effect controller160.

The media effect controller160may be configured to receive the image frame. The image in the image frame may be captured by the camera150. The media effect controller160may be configured to segment the image frame into a plurality of segments. The media effect controller160may be configured to determine a segment from the plurality of segments including a foreground region of a ROI and/or a background region of the ROI. The media effect controller160may be configured to detect whether the foreground region of the ROI and/or the background region of the ROI includes the motion information and the static information. The media effect controller160may be configured to automatically apply a motion type blur effect to the foreground region of the ROI and/or the background region of the ROI including the motion information. The media effect controller160may be configured to automatically apply a static type blur effect to the foreground region of the ROI and/or the background region of the ROI including the static information. The media effect controller160may be configured to generate the media file based on the applied the motion type blur effect and/or the static type blur effect. The media effect controller160may be configured to store the media file.

The media effect controller160may be configured to determine the foreground region of the ROI and/or the background region of the ROI of the image frame obtained by using a manual selection of a user of the electronic device100and/or AI model applied to the image frame.

The media effect controller160may be configured to enhance the color spectrum region for each segment of the plurality of segments. The media effect controller160may be configured to generate the color spectrum region for each segment. The media effect controller160may be configured to expand the wavelength range of the generated color spectrum region to enhance the value of each pixel of each segment.

The media effect controller160may be configured to obtain the wavelength value for each pixel of each segment, where each pixel includes an RGB value. The media effect controller160may be configured to categorize the obtained wavelength value based on the color spectrum region, where the color spectrum region ranging from a red color to violet color. The media effect controller160may be configured to determine the wavelength range and a wavelength range difference for each color for each segment, where the wavelength range may be determined between the lowest value of wavelength and the highest value of wavelength.

The media effect controller160may be configured to determine the typical wavelength range and the typical wavelength range difference for each color of each segment. The media effect controller160may be configured to determine the absolute difference between the determined wavelength range difference and the typical wavelength range difference for each color of each segment. The media effect controller160may be configured to determine the range change ratio for each color of each segment by dividing the absolute difference by the determined wavelength range. The media effect controller160may be configured to determine the first-new RGB value for each pixel of each segment based on the typical wavelength range difference, the determined wavelength range difference, the obtained wavelength value, and the lowest value of wavelength, where the first-new RGB value is enhanced value of the RGB value. The media effect controller160may be configured to average the range change ratio. The media effect controller160may be configured to determine the second-new RGB value for each pixel of each segment based on the average change ratio, where the second-new RGB value is the enhanced value of the first-new RGB value.

The media effect controller160may be configured to compare two consecutive image frames for obtaining the motion information and/or the static information, where the motion information includes a linear motion and/or a rotational motion. The linear motion is calculated based on a linear change in a position of each segment of the plurality of segments. The media effect controller160is configured to divide each segment of the plurality of segments of two consecutive image frames into four quads. The media effect controller160is configured to calculate the hash value based on the RGB values for each quad of each segment. The media effect controller160is configured to determine whether the hash value of at least two quads in two consecutive frames matches each other. The media effect controller160may be configured to detect the rotational motion in response to determining that the at least two quads match in the corresponding frames. The media effect controller160is configured to divide each segment of two consecutive image frames into four quads in response to determining that the at least two quads do not match in the corresponding frames.

The media effect controller160may be configured to apply the motion type blur effect on the foreground region of the ROI and/or the background region of the ROI when the foreground region of the ROI and/or the background region of the ROI contains the motion information i.e., motion value exceeding a predetermined threshold value is detected for the foreground region of the ROI and/or the background region of the ROI. The media effect controller160may be configured to apply a plurality of blur effect on the foreground region of the ROI and/or the background region of the ROI when the foreground region of the ROI and/or the background region of the ROI contains the static information.

The media effect controller160may be configured to calculate an absolute difference in the RGB value when moving from one pixel to another pixel in each sub-segment for each blur effect among the plurality of blur effects. The media effect controller160may be configured to apply an optimal blur effect among the plurality of blur effect based on the minimum change in color or the maximum change ratio associated with the calculated absolute difference.

The media effect controller160includes an image segmentor160a, a color spectrum controller160b, a motion detector160c, a blur generator160d, and an AI engine160e.

In an embodiment, the image segmentor160areceives the image frame and segments the image frame into the plurality of segments. Further, the image segmentor160adetermines the segment which includes either the foreground region of the ROI or the background region of the ROI. The foreground region of the ROI and the background region of the ROI of the image frame is obtained by using the manual selection of the user of the electronic device100and/or by applying an AI model (i.e., the AI engine160e) to the image frame.

The color spectrum controller160benhances the color spectrum region for each segment of the plurality of segments. The color spectrum controller160bgenerates the color spectrum region for each segment. The color spectrum controller160bexpands the wavelength range of the generated color spectrum region to enhance the value of each pixel of each segment. The color spectrum controller160bobtains the wavelength value for each pixel of each segment. The color spectrum controller160bcategorizes the obtained wavelength value based on the color spectrum region. Further, the color spectrum controller160bdetermines the wavelength range and the wavelength range difference for each color of each segment.

The color spectrum controller160bdetermines the typical wavelength range and the typical wavelength range difference for each color of each segment. The color spectrum controller160bdetermines the absolute difference between the determined wavelength range difference and the typical wavelength range difference for each color of each segment. The color spectrum controller160bdetermines the range change ratio for each color for each segment by dividing the absolute difference by the determined wavelength range. The color spectrum controller160bdetermines the first-new RGB value for each pixel of each segment based on the typical wavelength range difference, the determined wavelength range difference, the obtained wavelength value, and the lowest value of wavelength. The color spectrum controller160baverages the range change ratio. The color spectrum controller160bdetermines the second-new RGB value for each pixel of each segment based on the average change ratio.

The motion detector160cdetects whether the foreground region of the ROI and/or the background region of the ROI includes the motion information and the static information. The motion detector160ccompares two consecutive image frames for obtaining the motion information and/or the static information. The motion detector160ccalculates the linear motion based on the linear change in the position of each segment of the plurality of segments.

The motion detector160cdivides each segment of the plurality of segments of two consecutive image frames into four quads. The motion detector160ccalculates the hash value based on the RGB values for each quad. The motion detector160cdetermines whether the hash value of at least two quads matches the corresponding frames. The motion detector160cdetects the rotational motion in response to determining that the at least two quads match in the corresponding frames. The motion detector160cdivides each segment of two consecutive image frames into four quads in response to determining that the at least two quads do not match in the corresponding frames.

The blur generator160dautomatically applies the motion type blur effect on the foreground region of the ROI and/or the background region of the ROI includes the motion information and the static type blur effect on the foreground region of the ROI and/or the background region of the ROI includes the static information. The blur generator160dapplies the motion type blur effect on the foreground region of the ROI and/or the background region of the ROI when the foreground region of the ROI and/or the background region of the ROI contains the motion information. Further, the blur generator160dapplies the plurality of blur effect on the foreground region of the ROI and/or the background region of the ROI when the foreground region of the ROI and/or the background region of the ROI contains the static information.

The blur generator160dcalculates the absolute difference in the RGB value when moving from one pixel to another pixel in each sub-segment for each blur effect of the plurality of blur effect. The blur generator160dapplies the optimal blur effect from the plurality of blur effect based on the minimum change in color or the maximum change ratio associated with the calculated absolute difference. The blur generator160dgenerates the media file based on the applied the motion type blur effect and the static type blur effect. The blur generator160dstores the media file.

At least one of the plurality of modules/components may be implemented through the AI engine160e. A function associated with the AI engine160emay be performed through the memory110and the processor120. The one or a plurality of processors controls the processing of the input data in accordance with a predefined operating rule or the AI engine160estored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning.

Being provided through learning means that, by applying a learning process to a plurality of learning data, a predefined operating rule or the AI engine160eof the desired characteristic is made. The learning may be performed in the electronic device100itself in which AI according to an embodiment is performed, and/or maybe implemented through a separate server/system.

The AI engine (160e) may include a plurality of neural network layers. Each layer has a plurality of weight values and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

The learning process is a method for training a predetermined target device (e.g., a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning processes include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

AlthoughFIG.2shows various hardware components of the electronic device100, it is to be understood that other embodiments are not limited thereon. In other embodiments, the electronic device100may include fewer or more components. Further, the labels or names of the components are used only for illustrative purpose and does not limit the scope of the disclosure. One or more components can be combined together to perform same or substantially similar function to produce the media file with the blur effect.

FIG.3is a flow diagram300illustrating a method for producing the media file with the Bokeh effect, according to an embodiment of the disclosure. The operations302to320are performed by the electronic device100.

Referring toFIG.3, at operation302, the method includes receiving the image frame. At operation304, the image frame is segmented into the plurality of segments. At operation306, the segment is determined from the plurality of segments including the foreground region of the ROI and/or the background region of the ROI. At operation308, the color spectrum region for each segment of the plurality of segments is enhanced. The operation308may be skipped by a designer's choice in processing the input image.

At operations310to312, it is determined whether the foreground region of the ROI and/or the background region of the ROI includes the motion information and the static information. At operation314, the motion type blur effect is automatically applied on the foreground region of the ROI and/or the background region of the ROI when the foreground region of the ROI and/or the background region of the ROI includes the motion information. At operation316, the static type blur effect is automatically applied on the foreground region of the ROI and/or the background region of the ROI when the foreground region of the ROI and/or the background region of the ROI includes the static information. At operation318, the media file is generated based on the applied the motion type blur effect and the static type blur effect. At operation320, the media file is stored in a memory.

The various actions, acts, blocks, operations, or the like in the flow diagram300may be performed in the order presented, in a different order or simultaneously. In some embodiments, some of the actions, acts, blocks, operations, or the like may be omitted, added, modified, skipped, or the like without departing from the scope of the disclosure.

FIG.4is a flow diagram illustrating an example scenario where an electronic device determines one segment from the plurality of segments including the foreground region of the ROI and/or the background region of the ROI, according to an embodiment of the disclosure.

Referring toFIG.4, at operation401, the image segmentor160areceives the image frame. At operation402, the image segmentor160asegments the image frame into the plurality of segments S1-S5. For example, the image segmentor160auses a Deep Lab segmentation mechanism to segment the image frame into the plurality of segments S1-S5. At operations403and404, the image segmentor160adetermines the segment from the plurality of segments S1-S5 comprising the foreground region of the ROI (S1) and/or the background region of the ROI (S2-S5). The foreground region of the ROI (S1) and the background region of the ROI (S2-S5) is obtained by using the manual selection of the user of the electronic device100(user-selected object is considered the foreground region of the ROI (S1) in a manual mode) and/or the AI engine160eapplied to the image frame (the foreground region of the ROI (S1) is determined by using depth map analysis in an auto mode). That is, the foregoing region of the ROI may be a region with the lowest depth in the depth map of the captured image.

FIGS.5A,5B, and5Cillustrate a flow diagram illustrating an example scenario where a color spectrum controller160bof an electronic device100enhances the color spectrum region for each segment of the plurality of segments, according to an embodiment of the disclosure.

Referring toFIGS.5A,5B, and5C, at operation501, the color spectrum controller160breceives the foreground region of the ROI (S1) and the background region of the ROI (S2-S5) from the image segmentor160a. At operation502, the color spectrum controller160bobtaining the wavelength value for each pixel of each segment (S1-S5), where each pixel including the RGB value(s). For example, consider a pixel in a yellow area with RGB value(s) of (155,21,19), which is transformed to the wavelength value of (Wcurrent=640 nm) using an RGB spectrum conversion method. Calculate for each ROI segment (S1-S5) individually and assign each pixel to a relevant color in the spectrum.

At operation503, the color spectrum controller160bcategorizes the obtained wavelength value based on the color spectrum region, where the color spectrum region ranging from the red color to the violet color and determines the wavelength range and the wavelength range difference for each color for each segment, where the wavelength range including the lowest value of wavelength and the highest value of wavelength. The wavelength range and the wavelength range difference is calculated for each color for each segment (S1-S5) is given by the below equations,
Wavelength range old=(WoltoWou)  Equation 1
Wavelength range difference(Range_old)=Wou−WolEquation 2

In Equations 1 and 2, “W” indicates wavelength, “Wol” is W (old, lowest) indicates the lowest value of W for given color in the color spectrum, “Wou” is W (old, highest) indicates the highest value of W for given color in the color spectrum. For example, wavelength range calculation for the S1, as shown in Table 1 and wavelength range calculation for the S2, as shown in Table 2. If the range change is less than a certain number or is not detected in the particular ROI, the wavelength range obtained for the given color is discarded and set to 0.

At operation504, the color spectrum controller160bdetermines the typical wavelength range and the typical wavelength range difference for each color for each segment (S1-S5). Table 3 shows a typical spectrum range for colors in a visible range. The visible range is expanded to include the entire visible spectrum. The typical wavelength range difference is calculated for each color for each segment (S1-S5) is given by the below equation,
Wavelength range difference(Range_new)=Wnu−WnlEquation 3

In Equation 3, “Wnl” is W (new, lowest) indicates the lowest value of W for given color in the typical color spectrum, “Wnu” is W (new, highest) indicates the highest value of W for given color in the typical color spectrum.

At operation505, the color spectrum controller160bdetermines the absolute difference between the determined wavelength range difference and the typical wavelength range difference for each color for each segment (S1-S5). The absolute difference is calculated for each color for each segment (S1-S5) is given by the below equation,
Change in range=|Range_new−Range_old|  Equation 4

The color spectrum controller (160b) determines the range change ratio for each color for each segment by dividing the absolute difference by the determined wavelength range. The range change ratio is calculated for each color for each segment (S1-S5) is given by the below equation,
Range change ratio=(Change in range)/Range  Equation 5

For example, wavelength range broadening for the S1, as shown in Table 4, and wavelength range broadening for the S2, as shown in Table 5. Colors that are not noticed in the input range are disregarded and set to 0 during broadening.

At operation506, the color spectrum controller160bdetermines the first-new RGB value for each pixel of each segment (S1-S5) based on the typical wavelength range difference, the determined wavelength range difference, the obtained wavelength value, and the lowest value of wavelength, where the first-new RGB value is enhanced value of the RGB value. The first-new RGB value is calculated for each color for each segment (S1-S5) is given by the below equation,
Wnew=(Range_new/Range_old)*(Wcurrent−Wol)+WnuEquation 6

For example, the calculation for Wcurrent of 640 nm as per 502 for the S2 is as follows:
Wnew=(99/44)*(640−604)+601=682 nm.

The color spectrum controller160btransforms the calculated range to the first-new RGB value is (255, 9, 0); this calculation is applied to each pixel of each segment (S1-S5).

At operation507, the color spectrum controller160baverages the range change ratio. An average range change ratio is calculated for each color of each segment (S1-S5) is given by equation 7:
Average change ratio=(ΣRange change ratio for each color)/7  Equation 7

For example, the calculation for the segment S2 is as follows:
Average change ratio=(1.25+1.375+1.41)/7=0.576.

At operation508, the color spectrum controller160bdetermines the second-new RGB value for each pixel of each segment (S1-S5) based on the average change ratio, where the second-new RGB value is the enhanced value of the first-new RGB value. The second-new RGB value is calculated for each color for each segment (S1-S5) is given by equations 8 and 9:
G=min(255,g(1+δ)) for values above 128  Equation 8
G=max(0,g(1−δ)) for values below 128  Equation 9

In Equations 8 and 9, “δ” indicates the average change ratio, “G” indicates a new gray level for the channel and “g” indicates an old gray level for the channel.

For example, the calculation for the S2 is as follows:
R=255*(1+0.576)=min(401,255)=255
G=9*(1−0.576)=4
B=0*(1−0.576)=0

The second-new RGB value is (255, 4, 0); this calculation is applied on each pixel of each segment (S1-S5).

At operation509, the color spectrum controller160bgenerates the enhanced foreground region of the ROI (S1) and the enhanced background region of the ROI (S2-S5), and sends it to the motion detector160cfor further processing.

FIGS.6A,6B,6C, and6Dillustrate calculation and example scenarios where the motion detector160cof the electronic device100identifies whether the foreground region of the ROI and/or the background region of the ROI including motion information and/or static information present in the image frame, according to an embodiment of the disclosure.

Referring toFIG.6A, to identify that how motion is applied to the objects/segments of the image frame, the motion detector160ccalculates both the speed and direction of the motion in each segment (e.g., S1-S5). The motion detector160cutilizes successive frames (601,602) to identify the difference in the object position to calculate the difference in the position, and knowing frame rate can simply be utilized to compute the object's speed.

At operation603, consider an example for two consecutive frames where Fi (601) and F (i+1) (602) be the ithand (i+1)th frame at time ti and t (i+1). Then, the motion detector160cperforms object segmentation on the ROIs (e.g. S1-S5) to determine fine objects. Then, the motion detector160cdetermines all objects with the same type as in the previous frame (Fi601) within a distance Dmax of the object in frame F (i+1)602and the maximum speed of the objects moving in a direction is given as Smax. The motion detector160cdetermines the Dmax by Equations 9 and 10:
Δt=(t(i+1)−t(i))  Equation 10
Dmax=Smax/ΔtEquation 11

At operations604and605, the motion detector160cuses maximum overlapping607to identify which objects are closest to each other. This is determined via a simple intersection of areas. The area that is common to both objects determines the maximum overlapping areas for two objects. For example, A1, A2 be the area in a number of pixels of the objects in Frame Fi601and F (i+1)602. Let A3 be the area that is common to both. Then a percentage of the overlapping areas is given by Equation 12:
o=(A3/(A1+A2−A3))  Equation 12

The value of maximum o gives the object that is the same in both the frames. The objects with the maximum overlapping areas in the corresponding frames are considered same and are given same unique ID. The unique ID is a hash value that maps each object to a unique number in the frames. So the objects which are same will get the same hash in all the frames.

At operation606, the motion detector160ccalculates a linear displacement608by considering the position of one of the corners of the bounding box and then comparing the difference in pixel values. The motion detector160ccalculates the rotational motion by a Quadtree decomposition609, which is used to determine an amount of change in the pixel values around a pivot and then using that to determine an actual direction of motion and speed of change.

Referring toFIG.6B, an example scenario where the motion detector160cidentifies whether the motion information and/or the static information present in the image frame, the motion detector160ccomparing two consecutive image frames (610,611) for obtaining the motion information and/or the static information, as explained inFIG.6A. The motion information including the linear motion and/or the rotational motion. The motion detector160cuses a fine trained deep lab segmentation model to divide the segments (S1-S5) into finer detail segments.

At operations612and613, the motion detector160ccalculates the linear motion based on the linear change in the position of each segment of the plurality of segments (S1-S5). Then, the motion detector160ccalculates the rotational motion by dividing each segment of the plurality of segments (S1-S5) of two consecutive image frames into four quads (e.g. For Frame-1610: Ha1-Ha4, Frame-2611: Hb1-Hb4). Then, the motion detector160ccalculates the hash value based on the RGB values for each quad and determines whether the hash value of at least two quads matches the corresponding frames. If corresponding hashes are the same so no further Quadtree decomposition is required and no rotational change. Based on the preceding stages, the motion detector160crecognizes only the static information present in the current example situation.

Referring toFIG.6C, which illustrates another example scenario where the motion detector160cidentifies whether the motion information and/or the static information present in the image frame, the motion detector160ccompares two consecutive image frames (614,615) for obtaining the motion information and/or the static information, as explained inFIG.6A. The motion information including the linear motion and/or the rotational motion. The motion detector160cuses a fine trained deep lab segmentation model to divide the segments (e.g. car body, front tire, back tire, or the like) into finer detail segments624.

At operations616,617, and618, the motion detector160ccalculates the linear motion based on the linear change in the position of each segment of the plurality of segments (e.g. car body, front tire, back tire, or the like). Then, the motion detector160ccalculates the rotational motion by dividing each segment of the plurality of segments of two consecutive image frames into four quads. Then, the motion detector160ccalculates the hash value based on the RGB values for each quad and determines whether the hash value of at least two quads matches the corresponding frames. If corresponding hashes are not the same so further Quadtree decomposition is required and rotational change present in the corresponding quad. Based on the preceding stages, the motion detector160crecognizes the static information619and the motion information620present in the current example situation.

Referring toFIG.6D, a detailed example scenario is illustrated where the motion detector160ccalculates the motion information. At operation621, the motion detector160ccomparing two consecutive image frames (e.g. Frame (t−1) and Frame (t)) for obtaining the motion information. The motion detector160ccalculates the linear motion based on the linear change separation is given by (330−320, 320, 316)=(10, −4); this calculation estimate linear blur. At operation622, the motion detector160ccalculates the rotational motion by dividing each segment of the plurality of segments of two consecutive image frames into four quads. Then, the motion detector160ccalculates the hash value based on the RGB values for each quad and determines whether the hash value of at least two quads matches the corresponding frames. Here, in this example scenario, further decomposition is required as no areas are mapped.

At operation623, the motion detector160cagain divides each segment of the plurality of segments of two consecutive image frames into four quads. In this example scenario, no further decomposition is required as multiple (2 or more) areas have already been mapped. Further, the motion detector160ccalculates “0”, which represents the change in rotation from Frame (t−1) to Frame (t). A radial motion blur will be applied based on the value of the θ, in the direction of the motion (clockwise in this case). Based on the preceding stages, the motion detector160crecognizes the motion information (e.g. θ=0.52539 rad, clockwise and Linear separation=(10, −4)) present in the current example situation.

FIGS.7A,7B,7C, and7Dillustrate calculation and an example scenario where the blur generator of the electronic device automatically applies the motion type blur and/or static type blur effect on the foreground region of the ROI and/or the background region of the ROI, according to an embodiment of the disclosure.

Referring toFIG.7A, the blur generator160dapplies the linear motion blur based on the linear displacement and rotation calculation as explained inFIGS.6A-6D. The linear displacement can be easily calculated by considering the position of one of the corners of the bounding box and then comparing the difference in pixel values and speed of motion701is given by,
Speed of motion=D/ΔtEquation 13

The amount of linear motion blur applied is proportional to the speed of motion and is applied in direction of blur to the surroundings of the focused object. Further, the blur generator160dapplies the radial motion blur based on the Quadtree decomposition, as explained in theFIGS.6A,6B,6C, and6D, with increased depth is used to determine the exact change in angle702, the direction can be a clockwise and/or an anti-clockwise, amount of speed is given by,
Speed=Θ/ΔtEquation 14
Where, Θ=(change in angle), the amount of radial blur applied is proportional to the amount of speed.

At operation703, example scenario for gradient-based blur identification by the blur generator160d:

Each blur is applied to separate layer depth maps as shown inFIG.703.

This will be used in the next operation when the gradient ratio for blur type estimation.

For each background layer in the depth map:

Variance (σ) for filters is determined as linearly increasing function.

The size of a kernel is determined as 2*ceil (2σ)+1 (for disk averaging, the radius is taken as ceil (2σ)). The reason for taking size as such is based on the Gaussian kernel. The Gaussian kernel values become negligible after this size towards the border.

The image pixels corresponding to the background layer are blurred.

Furthermore, the blur generator160destimates per pixel Gradient ratio, Gradient is the change in value when moving from one pixel to another. Consider, Dorig is gradient for the original image and pixel p and Dbi is gradient for blur type i for pixel p. The blur generator160dcalculates Dorig/Dbi, which gives a ratio of the gradient of the original image to the blurred image. Furthermore, the blur generator160ddetermines the blur type based on a max (Dorig/Dbi). The max (Dorig/Dbi) gives a minimum change in color when moving from a nearby pixel towards the target pixel/current pixel. Furthermore, the blur generator160ddetermines gradient value for Pixel P is calculated using a root mean square formula as defined in Equation 15:

D⁡(x,y)=12⁢(∑cy={-s,s},cx={-s,s}(p⁡(x,y)-p⁡(x+cx,y+cy))1/2

In Equation 15, D(x, y) is the Gradient at pixel (x, y) and s is the operation size of the gradient. Usually, the value of s will be 3-10. But different values can be used based on the depth layer and using more advanced object-defining algorithms. E.g. for larger objects, a larger s can be used for better blurring, while smaller values will work when using for smaller objects.

Furthermore, the blur generator160doptimizes a similar blur region by calculating the distance between two-pixel values as follows:
Difference=|Target(R,G,B)−Nearby(R,G,B)|=[|Target(R)−Nearby(R)|, |Target(G)−Nearby(G)|,|Target(B)−Nearby(B)|]=Difference(R),Difference(G), Difference(B)  Equation 16
Distance=Max(Difference(R),Difference(G),Difference(B))  Equation 17

The blur generator160ddetermines if (Distance <17), the pixel is considered to be in the same segment and otherwise, the pixel is ignored; calculation of threshold “17”703ais calculated by: Total number of colors is 16777216 (2{circumflex over ( )}24), number of distinguishable colors by a human eye is 10,000,000, Number of indistinguishable colors is 6777216. Since there are 3 channels Red, Green, Blue, therefore each channel contributes around 2259072 (6777216/3) colors which are approximately 13.46% (2259072/16777216). Since a single channel value varies from 0-255 therefore 0.1346*255=34.323˜35 hence 35 consecutive values are approximately indistinguishable. Hence, around a target of 17 values on each side can be considered similar.

Referring toFIGS.7B and7C, consider the example of the yellow bounded region all the static blurs are applied to this region. At operations704and705, the blur generator160dapplies the plurality of blur effect (e.g. Median blur, a Bilateral blur, a Gaussian blur, a Box blur, a Channel blur) on the foreground region of the ROI (S1) and/or the background region of the ROI (S2-S5) when the foreground region of the ROI (S1) and/or the background region of the ROI (S2-S5) contains the static information.

At operations706and707, the blur generator160dcalculates the absolute difference (|Vs−Ve|) in the RGB value when moving from one pixel to another pixel in each sub-segment (S1-S5) for each blur effect of the plurality of blur effect. Then, the blur generator160dapplies the optimal blur effect (e.g. Median blur) from the plurality of blur effect based on the minimum change (e.g. D=Root Mean Square (RMS)=19.6) in color or the maximum change ratio (Dorg/Di) associated with the calculated absolute difference. For example, consider the row2highlighted by rectangle for median blur: Absolute difference=|(13, 14, 15)−(32, 15, 18)|=(|13−32|, |14−15|, |15−18|)=(19, 1, 3) and RMS is calculated by Dorg/Di=(133/19.26)=6.9. Dorg is RMS value for the original image and Di is the RMS value for the blur. For example, Pixel values for all the blurs can be summarized and blur selection for the yellow bounded region, as shown in Table 6, this calculation is done for each pixel of each segment by the blur generator (160d) which contains the static information. Table 7 and Table 8 shows why the media blur is selected based on the calculation above.

TABLE 7RGB for nearbyRGB for targetBlur Typepixel (Vs)Pixel (Ve)Original Image15, 15, 17146, 34, 34Median Blur13, 14, 1532, 15, 18Bilateral Blur15, 15, 1759, 21, 19Gaussian Blur15, 15, 1782, 25, 25Box Blur15, 15, 17106, 23, 15Channel Blur16, 17, 17111, 26, 21

TABLE 8Absolute Difference (Ve − Vs)RMS (D)Dorg/Di131, 19, 17133119, 1, 319.266.944, 6, 244.452.9967, 10, 868.211.9491, 8, 291.371.45595, 9, 495.511.39

Based on Dorg/Di in Table 8, the Median blur is selected as optimal blur.

At operation708, since nearby pixels have almost similar RGB values optimization is done by selecting all the nearby pixels which have the RGB pixel values within the range of the original target pixel. The area is constructed by considering all the pixels within the range of the target pixel which have a threshold of less than 17 pixels for a single channel. An example of the calculations by the blur generator160d: Target pixel is (146, 34, 34) and nearby pixel is (146, 42, 29), Absolute difference=(|146−146|, |34−42|, |34−29|) is (0, 8, 5) and Channel distance is Max (0, 8, 5)=8. Since the result value 8 is less than 17 (8<17), the pixel is considered to be in the same segment. At operation709, the above mention calculation is done for each pixel of each segment by the blur generator (160d) which contains the static information, and based on the calculation the blur generator (160d) selects/applies the optimal blur effect on a particular region of the segment. At operation710, the blur generator160dgenerates the multi-type blurring effect in real-time in the media file (e.g. image frame), enhancing the user's experience.

FIGS.8A and8Billustrate an example flow diagram for producing the media file with the Bokeh effect and a comparison between the Bokeh effect of the existing method/electronic device and the Bokeh effect of the proposed method/electronic device, according to an embodiment of the disclosure.

Referring toFIG.8A, the example flow diagram for producing the media file with the Bokeh effect. The technical description of operations801ato807ais already explained inFIGS.2to4,5A to5C,6A to6D,7A, and7B.

Referring toFIG.8B, at operation801b, details in the ROI's foreground region are significantly clearer and more noticeably separated from the ROI's background region in the proposed bokeh effect. At operation802b, the proposed bokeh effect reduces unwanted contrasting features while keeping general background object details. At operation803b, segmentation along with ROI separation helps to minimize the unwanted details in the Non-ROI background regions.

The proposed bokeh effect in operation804aprovides color amplification in the foreground region and range extension may be seen. Further improvement may be seen in contrasting sections of the background region when applying numerous blurs. As a result of this blurring effect, the foreground is distinct and the background is smoothed. Grains were visible in the image background wall earlier which were smoothened in a final image, as can be seen in reference numerals801b,802b, and803b.

The proposed method enhances the bokeh effect where the images are segmented in operation805a(i.e., foreground region and background region are separated and objects are extracted from the background one by one to find ROI). After that, each region is analyzed, and based on the gradient difference different types of blur are applied to different objects in operation806a. The foreground region is also analyzed to check if the object is in motion and accordingly the different types of motion blur i.e. radial or motion type blur are applied to the foreground region and surrounding region so that the object appears in motion. Finally, in operation807a, all the objects are merges and a Gaussian smoothening is applied on the background region so that the area near the boundaries of different segments appears as one unit and not different segments.

FIGS.9A and9Billustrate example scenarios for real-time Bokeh effect generation for videos and a ring effect correction due to multiple light reflections on a camera of the electronic device according to an embodiment of the disclosure.

Referring toFIG.9A, in the method/electronic device for live focus videos according to the related art, the bokeh effect does not change automatically based on the background region. There are few filters available, but the user needs to change the filters manually based on the background region, and if the background region changes (for example, an object begins to move), then the filters do not work effectively and the user needs to change the filter. Also, the blur intensity is not suggested and the user needs to update the blur intensity based on background. While the proposed method (901-902) automatically applies the motion type blur effect on the foreground region of the ROI and/or the background region of the ROI that contains motion information and the static type blur effect on the foreground region of the ROI and/or the background region of the ROI that contains static information. As a result, the multi-type blurring effect is created in real-time in the media file, enhancing the user's experience.

Referring toFIG.9B, sometimes when the image contains a bright source light, as in image903, it may be possible that sometimes ring effect will be observed due to internal reflection in the camera with rings around an object. The proposed method can be used to correct this effect. The original image903is passed through the AI engine160etrained to detect this effect. During the ROI separation region. The resulting image904is passed further down the pipeline until the blur is applied dynamically using the proposed method. The proposed method reduces the effect of the ring in the input image905. The input image is then merged with the original image while the corrected colors from the blur applied will be used to correct the effect.

The embodiments disclosed herein can be implemented using at least one hardware device and performing network management functions to control the elements.

The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation.