Patent Description:
Recently, electronic devices have been providing more diversified services and additional functions. Various applications executable on electronic devices are being developed to meet the diverse demand of users and to raise the utility of electronic devices.

Camera applications are among these various applications, and a user may take a selfie or background using the camera of their electronic device. The electronic device may include a camera module for capturing images. The camera module may typically include a lens for collecting light, a photodiode for converting the collected light into an electrical signal, and an analog-to-digital converter (ADC) for converting the electrical signal, which is an analog signal, into a digital electrical signal. The process of a camera module that converts electrical signals from multiple photodiodes into digital electrical signals and outputs the digital electrical signals may be called 'read-out'.

<CIT> discloses a method for switching from a low output frame rate to a higher output frame rate based on the camera motion and an object motion exceeding different thresholds. <CIT> discloses a method for adapting the frame rate of a surveillance camera wherein the frame rate is increased if the pixel change value exceeds a given threshold. <CIT> discloses a surveillance system wherein the frame rate of a camera is increased or decreased based on comparing a pixel change metric to a threshold.

Super slow motion video recording uses dedicated equipment with a superior processing capability and large storage. Portable electronic devices are subject to significant limitations in super slow motion video recording due to limitations of their memory and processor. Upon recording video in super slow motion under the same conditions without considering the state of the electronic device, the electronic device would fail to produce a super slow motion video in the user's desired quality. When the electronic device is set to begin recording super slow motion video based on the movement of an object in the region of interest (ROI), the electronic device may determine its own movement as movement of the object in the ROI, erroneously initiating super slow motion recording.

According to an embodiment, an electronic device according to claim <NUM> is provided.

According to an embodiment, a method for operating an electronic device according to claim <NUM> is provided.

Definitions for certain words and phrases are provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

According to an embodiment, an electronic device may set the conditions for super slow motion video recording based on illuminance information and movement information about the surroundings of the electronic device and perform super slow motion video recording under the conditions. According to an embodiment, an electronic device may identify the movement of an object in an ROI in an image frame or an object in the background region outside the RIO and perform super slow motion video recording.

A more complete appreciation of the disclosure and many of the attendant aspects thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:.

The electronic device <NUM> may communicate with the electronic device <NUM> via the server <NUM>. According to an embodiment, the electronic device <NUM> includes a processor <NUM>, and a memory <NUM>, and may include an input device <NUM>, a sound output device <NUM>, a display device <NUM>, an audio module <NUM>, includes a sensor module <NUM>, and may include an interface <NUM>, a haptic module <NUM>, includes a camera module <NUM>, and may include a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module (SIM) <NUM>, or an antenna module <NUM>. In some embodiments, at least one (e.g., the display device <NUM>) of the components may be omitted from the electronic device <NUM>, or one or more other components may be added in the electronic device <NUM>.

The sensor module <NUM> may detect an operational state (e.g., power or temperature) of the electronic device <NUM> or an environmental state (e.g., a state of a user) external to the electronic device <NUM>, and then provide an electrical signal or data value corresponding to the detected state.

<FIG> is a block diagram <NUM> illustrating the camera module <NUM> (e.g., the camera module <NUM> of <FIG>) according to various embodiments. Referring to <FIG>, the camera module <NUM> may include a lens assembly <NUM>, a flash <NUM>, an image sensor <NUM>, an image stabilizer <NUM>, memory <NUM> (e.g., buffer memory), or an image signal processor <NUM>. The lens assembly <NUM> may collect light emitted or reflected from an object whose image is to be taken. The lens assembly <NUM> may include one or more lenses. According to an embodiment, the camera module <NUM> may include a plurality of lens assemblies <NUM>. In such a case, the camera module <NUM> may form, for example, a dual camera, a <NUM>-degree camera, or a spherical camera. Some of the plurality of lens assemblies <NUM> may include the same lens attribute (e.g., view angle, focal length, auto-focusing, f number, or optical zoom), or at least one lens assembly may include one or more lens attributes different from those of another lens assembly. The lens assembly <NUM> may include, for example, a wide-angle lens or a telephoto lens.

The flash <NUM> may emit light that is used to reinforce light reflected from an object. According to an embodiment, the flash <NUM> may include one or more light emitting diodes (LEDs) (e.g., a red-green-blue (RGB) LED, a white LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or a xenon lamp. The image sensor <NUM> may obtain an image corresponding to an object by converting light emitted or reflected from the object and transmitted via the lens assembly <NUM> into an electrical signal. According to an embodiment, the image sensor <NUM> may include one selected from image sensors having different attributes, such as a RGB sensor, a black-and-white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor <NUM> may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer <NUM> may move the image sensor <NUM> or at least one lens included in the lens assembly <NUM> in a particular direction, or control an operational attribute (e.g., adjust the read-out timing) of the image sensor <NUM> in response to the movement of the camera module <NUM> or the electronic device <NUM> including the camera module <NUM>. This allows compensating for at least part of a negative effect (e.g., image blurring) by the movement on an image being captured. According to an embodiment, the image stabilizer <NUM> may detect such a movement by the camera module <NUM> or the electronic device <NUM> using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module <NUM>. According to an embodiment, the image stabilizer <NUM> may be implemented, for example, as an optical image stabilizer. The memory <NUM> may store, at least temporarily, at least part of an image obtained via the image sensor <NUM> for a subsequent image processing task. For example, if image capturing is delayed due to shutter lag or multiple images are quickly captured, a raw image obtained (e.g., a Bayer-patterned image, a high-resolution image) may be stored in the memory <NUM>, and its corresponding copy image (e.g., a low-resolution image) may be previewed via the display device <NUM>. Thereafter, if a specified condition is met (e.g., by a user's input or system command), at least part of the raw image stored in the memory <NUM> may be obtained and processed, for example, by the image signal processor <NUM>. According to an embodiment, the memory <NUM> may be configured as at least part of the memory <NUM> or as a separate memory that is operated independently from the memory <NUM>.

The image signal processor <NUM> may perform one or more image processing with respect to an image obtained via the image sensor <NUM> or an image stored in the memory <NUM>. The one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesizing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor <NUM> may perform control (e.g., exposure time control or read-out timing control) with respect to at least one (e.g., the image sensor <NUM>) of the components included in the camera module <NUM>. An image processed by the image signal processor <NUM> may be stored back in the memory <NUM> for further processing, or may be provided to an external component (e.g., the memory <NUM>, the display device <NUM>, the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) outside the camera module <NUM>. According to an embodiment, the image signal processor <NUM> may be configured as at least part of the processor <NUM>, or as a separate processor that is operated independently from the processor <NUM>. If the image signal processor <NUM> is configured as a separate processor from the processor <NUM>, at least one image processed by the image signal processor <NUM> may be displayed, by the processor <NUM>, via the display device <NUM> as it is or after being further processed.

According to an embodiment, the electronic device <NUM> may include a plurality of camera modules <NUM> having different attributes or functions. In such a case, at least one of the plurality of camera modules <NUM> may form, for example, a wide-angle camera and at least another of the plurality of camera modules <NUM> may form a telephoto camera. Similarly, at least one of the plurality of camera modules <NUM> may form, for example, a front camera and at least another of the plurality of camera modules180 may form a rear camera.

<FIG> is a block diagram illustrating a structure of an image sensor according to an embodiment of the disclosure.

According to an embodiment of the disclosure, an image sensor <NUM> may be a component of a camera module (e.g., <NUM> or <NUM>) in an electronic device (e.g., <NUM>).

Referring to <FIG>, according to an embodiment, the image sensor <NUM> (e.g., the image sensor <NUM>) may include at least one of a pixel array <NUM>, a row-driver <NUM>, a column-readout circuit <NUM>, a controller <NUM>, a memory <NUM>, or an interface <NUM>.

The pixel array <NUM> may include a plurality of pixels <NUM> to <NUM>. For example, the pixel array <NUM> may include a structure in which the plurality of pixels <NUM> to <NUM> are arrayed in an MxN matrix pattern (where M and N are positive integers). The pixel array <NUM> where the plurality of pixels <NUM> to <NUM> are arrayed in a two-dimensional (2D) MxN pattern may include M rows and N columns. The pixel array <NUM> may include a plurality of photosensitive elements, e.g., photodiodes or pinned photodiodes. The pixel array <NUM> may detect light by using the plurality of photosensitive elements and convert the light into an analog electrical signal to provide an image signal.

The row-driver <NUM> may drive the pixel array <NUM> for each row. For example, the row-driver <NUM> may output transmission control signals to the transmission transistors of the plurality of pixels <NUM> to <NUM> in the pixel array <NUM>, reset control signals to control reset transistors, or selection control signals to control selection transistors to the pixel array <NUM>. The row-driver <NUM> may identify a row to be read out.

The column-readout circuit <NUM> may obtain analog electrical signals provided by the pixel array <NUM>. For example, the column-readout circuit <NUM> may obtain an analog electrical signal from a column line selected among the plurality of columns constituting the pixel array <NUM>. The column-readout circuit <NUM> may include an analog-digital converter (ADC) <NUM> that may convert the analog electrical signal obtained from the selected column line into pixel data (or a digital signal) and output the pixel data. Meanwhile, the column-readout circuit <NUM> that obtains an analog electrical signal from the pixel array <NUM>, converts the obtained analog electrical signal into pixel data by using the ADC <NUM>, and outputs the pixel data may be referred to as read-out. The column-readout circuit <NUM> and the ADC <NUM> may identify a column to be read out.

According to an embodiment of the disclosure, the column-readout circuit <NUM> of the image sensor <NUM> that supports slow motion recording may include a plurality of ADCs <NUM>. Each of the plurality of ADCs <NUM> may be connected in parallel with a respective one of the plurality of photodiodes in the pixel array <NUM>, and analog electrical signals simultaneously obtained from the plurality of photodiodes may quickly be converted into pixel data based on the parallel structure. The column-readout circuit <NUM> of the image sensor <NUM> that supports slow motion recording may perform a read-out at a high frame rate (e.g., <NUM> frames per second (fps)). For example, reading out at <NUM> fps means that obtaining an analog electrical signal from the pixel array <NUM>, converting the obtained analog electrical signal into pixel data by using the ADC <NUM>, and outputting the pixel data are performed once every <NUM>/<NUM> seconds. In other words, reading out at 960fps may mean that <NUM> image frames are outputted per second.

The controller <NUM> may obtain an image frame based on the pixel data obtained from the column-readout circuit <NUM>. The controller <NUM> may output the image frame through the interface <NUM> to an external circuit <NUM> (e.g., an image signal processor (ISP), processor, communication circuit, or external server). According to an embodiment of the disclosure, the controller <NUM> may provide transmission control signals to control the transmission transistors of the plurality of pixels <NUM> to <NUM>, reset control signals to control reset transistors, or selection control signals to control selection transistors and provide the signals to the row-driver <NUM>. The controller <NUM> may provide a selection control signal to select at least one column line from among the plurality of column lines constituting the pixel array <NUM> and provide the signal to the column-readout circuit <NUM>. For example, the column-readout circuit <NUM> may enable some of the plurality of column lines and disable the other column lines based on selection control signals provided from the controller <NUM>. The controller <NUM> may be implemented in a processor (e.g., <NUM>) including a central processing unit (CPU) or application processor (AP), a sort of block or module. When the controller <NUM> is implemented as a block, the controller <NUM> may include a subtractor for detecting the difference between, e.g., images, or a comparator for comparing images. According to an embodiment of the disclosure, the controller <NUM> may downsize read-out images and compare the plurality of downsized images to detect differences between the images.

The memory <NUM> may include a volatile and/or non-volatile memory. The memory <NUM> is a storage device inside the image sensor <NUM>. The memory <NUM> may include a buffer memory. According to an embodiment of the disclosure, the memory <NUM> may temporarily store digital signals output from the column-readout circuit <NUM> or the controller <NUM>. For example, the memory <NUM> may include at least one image frame obtained based on light obtained by the pixel array <NUM>. The memory <NUM> may store at least one digital signal obtained from the external circuit <NUM> through the interface <NUM>.

According to an embodiment of the disclosure, the memory <NUM> may store at least one image frame read out at an Nth frame rate (e.g., <NUM> fps) or an Mth frame rate (e.g., 120fps) from the column-readout circuit <NUM> and deliver at least one image frame stored through the interface <NUM> to the external circuit <NUM> (e.g., an IPS, processor, communication circuit, or external server). In other words, the memory <NUM> may store at least one image frame read out once every <NUM>/<NUM> seconds or every <NUM>/<NUM> seconds from the column-readout circuit <NUM>, and the memory <NUM> may deliver at least one image frame stored through the interface <NUM> to the external circuit <NUM>. The speed at which the image frame is transferred to the external circuit <NUM> is not limited thereto. According to an embodiment, the electronic device <NUM> may transfer immediately, without storing, the read-out image frame through the interface <NUM> to the external circuit <NUM>.

Meanwhile, the controller <NUM> may store only some of N image frames read out through the column-readout circuit <NUM> at the Nth frame rate (e.g., <NUM> fps) in the memory <NUM>, allowing for substantially the same effect as how M image frames are obtained, which are read out at the Mth frame rate (e.g., <NUM> fps). For example, the controller <NUM> may store only one of eight image frames read out at 960fps for <NUM>/<NUM> seconds in the memory <NUM>. When a plurality of image frames read out at 960fps, only image frames selected in a <NUM>:<NUM> ratio are stored in the memory <NUM>, the image frames stored in the memory <NUM> may be substantially the same image frames as those read out at 120fps through the column-readout circuit <NUM>. For example, when a video constituted of only image frames obtained at the cycle of <NUM>/<NUM> seconds is defined as '<NUM> fps video,' a video constituted of only image frames selected in a <NUM>:<NUM> ratio from among the plurality of image frames read out at 960fps may be defined as a 120fps video. A video constituted of only image frames read out at 120fps through the column-readout circuit <NUM> may also be defined as a 120fps video.

The interface <NUM> may include, e.g., the interface <NUM> or the communication module <NUM>. The interface <NUM> may connect components of the image sensor <NUM>, e.g., the controller <NUM> or the memory <NUM>, with the external circuit <NUM> in a wireless or wired scheme. For example, the interface <NUM> may deliver at least one image frame stored in the memory <NUM> of the image sensor <NUM> to the external circuit <NUM>, e.g., the memory (e.g., <NUM>) of the electronic device (e.g., <NUM>). The interface <NUM> may also deliver control signals from the processor (e.g., <NUM>) of the electronic device (e.g., <NUM>) to the controller <NUM> of the image sensor <NUM>.

According to an embodiment of the disclosure, the image sensor <NUM> may communicate with the external circuit <NUM> through the interface <NUM>, e.g., in a serial communication scheme. For example, the memory <NUM> of the image sensor <NUM> may communicate with the processor (e.g., <NUM>) of the electronic device (e.g., <NUM>) in an inter-integrated circuit (I<NUM>C) scheme.

According to an embodiment of the disclosure, the image sensor <NUM> may connect with the external circuit <NUM> through the interface <NUM>, e.g., an interface as defined based on the mobile industry processor interface (MIPI) protocol. For example, the memory <NUM> of the image sensor <NUM> may communicate with the processor (e.g., <NUM>) of the electronic device (e.g., <NUM>) based on the interface defined in the MIPI protocol. The interface <NUM>, e.g., the interface defined based on the MIPI protocol, may deliver pixel data corresponding to the image frames stored in the memory <NUM> to the external circuit <NUM> at the cycle of <NUM>/<NUM> seconds.

Meanwhile, while the image frames stored in the memory <NUM> are delivered through the interface <NUM> having an output speed of 240fps to the external circuit <NUM> once every <NUM>/<NUM> seconds, at least some of the image frames read out in real-time through the column-readout circuit <NUM> may be delivered to the external circuit <NUM> as preview images once every <NUM>/<NUM> seconds. The processor <NUM> in the external circuit <NUM> may display, through the display, all or some of the image frames output as preview images from the image sensor <NUM> at 30fps or 60fps.

All or some of the above-described components <NUM> to <NUM> may be included in the image sensor <NUM>, and each component may be configured in a single unit or multiple units. The frame rates 120fps, 240fps, and 960fps, used in the above embodiments may vary depending on the settings of the electronic device or the performance of the interface.

<FIG> is a block diagram illustrating a process for obtaining an image frame through an image sensor according to an embodiment. An image sensor <NUM> may be a component of a camera module (e.g., <NUM> or <NUM>) in an electronic device (e.g., <NUM>).

Referring to <FIG>, the image sensor <NUM> may include at least one a pixel array <NUM>, a memory <NUM>, and an interface <NUM>. The image sensor <NUM> may include the whole or part of the image sensor <NUM> of <FIG>.

The pixel array <NUM> of the image sensor <NUM> may output an electrical signal corresponding to light obtained from the outside. For example, the pixel array <NUM> may include a plurality of pixels constituted of photodiodes. The photodiodes may obtain light and provide analog electrical signals corresponding to the obtained light. Analog electrical signals provided from the plurality of photodiodes constituting the plurality of pixels may be converted into a plurality of pieces of pixel data through a column-readout circuit (e.g., <NUM>). In this case, each piece of pixel data may mean a pixel value corresponding to its respective pixel. A set of a plurality of pieces of pixel data obtained at a particular time may constitute at least one image frame.

According to an embodiment of the disclosure, the pixel array <NUM> of the image sensor <NUM> may output a plurality of image frames <NUM> to <NUM> at a preset read-out speed. For example, when the read-out speed is set to 960fps, the image sensor <NUM> may read-out <NUM> image frames every second based on light obtained by the pixel array <NUM>. According to an embodiment, the electronic device <NUM> may detect an event of slow motion recording while recording in a normal mode, in which case the electronic device <NUM> may turn the read-out rate from 120fps to 950fps.

The plurality of image frames <NUM> to <NUM> read out may be stored in a memory <NUM> inside the image sensor <NUM>. According to an embodiment of the disclosure, the memory <NUM> of the image sensor <NUM> may include a buffer memory <NUM>. For example, some of the plurality of image frames <NUM> to <NUM> read out at 960fps may be stored in the buffer memory <NUM>. Among a plurality of image frames continuously read out, a predetermined number of image frames may be stored in the buffer memory <NUM>. The processor (e.g., <NUM> or the controller <NUM>) may repeat the operations of deleting the image frame stored earliest from among the image frames stored in the buffer memory <NUM> and storing the image frame latest from among the image frames.

At least one image frame stored in the memory <NUM> of the image sensor <NUM> may be delivered to an external circuit <NUM> through an interface <NUM> (e.g., <NUM>). For example, the processor (e.g., <NUM> or the controller <NUM>) may control the interface <NUM> to deliver at least one image frame stored in the memory <NUM> to the external circuit <NUM>.

According to an embodiment, the image sensor <NUM> may exclude the buffer <NUM>. The image frames <NUM>, <NUM>, <NUM>, and <NUM> from the pixel array <NUM> may directly be transferred through the interface <NUM> to the external circuit <NUM>.

<FIG> is a block diagram illustrating an electronic device <NUM> according to an embodiment of the disclosure. An electronic device <NUM> may include a processor <NUM>, a display <NUM>, a sensor circuit <NUM>, a camera module <NUM>, and a memory <NUM>.

According to an embodiment, the processor <NUM> (e.g., the processor <NUM> of <FIG>) may control the display <NUM> (e.g., the display device <NUM>), the sensor circuit <NUM> (e.g., the sensor module <NUM>), the camera module <NUM> (e.g., the camera module <NUM> or <NUM>), and the memory <NUM> (e.g., the memory <NUM>) connected with the processor <NUM> and perform various types of data processing or computation. For example, the processor <NUM> may obtain sensor data from the sensor circuit <NUM> and identify a reference threshold for slow motion recording using the sensor data. The processor <NUM> may obtain a slow motion video in response to an object (e.g., one in an image) moving at the threshold by using the camera module <NUM>. The processor <NUM> may display on the display <NUM>, or store in the memory <NUM>, a plurality of image frames constituting the slow motion video.

According to an embodiment, the display <NUM> (e.g., the display device <NUM> of <FIG>) may display at least one image frame obtained through the image sensor <NUM> (e.g., the image sensor <NUM> of <FIG>). For example, the display <NUM> may display the preview image provided through the image processor <NUM> or at least one video stored in the memory <NUM>.

According to an embodiment, the sensor circuit <NUM> (e.g., the sensor module <NUM> of <FIG>) may detect the movement of the electronic device <NUM> by using at least one sensor (e.g., a gyro sensor or an acceleration sensor). The sensor circuit <NUM> may transfer data regarding the movement of the electronic device <NUM> to the processor <NUM> or the camera module <NUM>.

According to an embodiment, the camera module <NUM> (e.g., the camera module <NUM> of <FIG> or the camera module <NUM> of <FIG>) may include an image sensor <NUM> (e.g., the image sensor <NUM>), an image processor <NUM> (e.g., the image signal processor <NUM>), and an encoder <NUM>. Meanwhile, at least any one of the components <NUM>, <NUM>, and <NUM> in the camera module <NUM> may be designed in hardware, software, firmware, or in other various forms, and each component may be termed with a different name.

According to an embodiment, the image processor <NUM> may obtain at least one piece of pixel data output from the image sensor <NUM> and process or edit the obtained pixel data for delivery to at least one component of the electronic device. The image processor <NUM> may resize the image frame corresponding to the pixel data obtained from the image sensor <NUM> to be displayed through the display <NUM>. "Preview image" means an image that is displayed on the display <NUM> when the user takes an image of at least one object using the camera module <NUM> of the electronic device <NUM>. At least one image frame obtained through the image sensor <NUM> may be displayed in real-time on the display <NUM> as a preview image, and the user of the electronic device may easily take images of the external object through the preview image.

According to an embodiment, the encoder <NUM> may provide at least one video based on the pixel data processed by the image processor <NUM>. For example, the encoder <NUM> may compress an image frame corresponding to the pixel data obtained from the image sensor <NUM> to be stored in the memory <NUM>. Further, the encoder <NUM> may encode image frames corresponding to a plurality of pieces of pixel data obtained from the image sensor <NUM>. The encoder <NUM> may array the plurality of image frames in the order obtained through the image sensor <NUM> based on the plurality of pieces of pixel data obtained from the image sensor <NUM>. The processor <NUM> may store the video encoded by the encoder <NUM> in the memory <NUM>.

According to an embodiment, at least one of the image processor <NUM> or the encoder <NUM> may be included not in the camera module <NUM>, but instead in the processor <NUM>.

According to an embodiment, the memory <NUM> (e.g., the memory <NUM>) may store at least one image frame obtained through the image sensor <NUM> or at least one video encoded through the encoder <NUM>. Meanwhile, the image sensor <NUM> may include at least one memory (e.g., <NUM>) in which case the memory <NUM> may mean a different storage space positioned in a place separated from the memory <NUM>.

<FIG> is a view illustrating a method for performing slow motion recording using an electronic device <NUM> according to an embodiment of the disclosure. The electronic device <NUM> may include an image sensor <NUM>, includes a processor (e.g., <NUM>), and may include a display (e.g., <NUM>). Here, the image sensor may include at least one of a pixel array <NUM>, a row-driver <NUM>, a column-readout circuit <NUM>, a controller <NUM>, a memory <NUM>, and an interface <NUM>.

Referring to <FIG>, the processor may obtain a plurality of image frames for an external object <NUM> through the image sensor <NUM>, provide a video by using at least some of the obtained plurality of image frames <NUM>, and in response to receiving a request to play video, display a video screen <NUM>. The video may be configured as a 120fps video or 960fps video depending on the time when each of the image frames constituting the video is obtained and the read-out speed of the image sensor. Here, the 120fps video means that the image frames constituting the video are obtained at the cycle of <NUM>/<NUM> seconds, and the 960fps video means that the image frames constituting the video are obtained at the cycle of <NUM>/<NUM> seconds. The video screen <NUM> may include content <NUM> indicating the current position of playback.

According to an embodiment of the disclosure, the controller <NUM> included in the image sensor may set the read-out speed, which is the output speed of the column-readout circuit <NUM>, as 120fps or 960fps, corresponding to a request related to the execution of a slow motion recording mode. When the read-out speed is set to 120fps or 960fps, the column-readout circuit <NUM> may read-out <NUM> or <NUM> image frames every second based on light obtained by the pixel array <NUM>.

The controller <NUM> may identify a pixel variation in the image frame by using at least two of the plurality of image frames read out. For example, the controller <NUM> may detect a pixel value variation in a particular region (e.g., at least one of the ROI or background region) based on at least two of the plurality of image frames read out through the column-readout circuit <NUM> before a first time <NUM>. For example, the pixel value variation sensible through the controller <NUM> may include a variation based on at least one of the movement of the electronic device <NUM> between two image frames, the illuminance of the outside of the electronic device <NUM>, or the size of the ROI. The controller <NUM> may identify a threshold for initiating slow motion recording based on the detected pixel value variation. Specifically, as described below, the threshold may be identified based on at least one of the movement of the electronic device <NUM>, the illuminance of the outside of the electronic device <NUM>, or the size of the ROI. The controller <NUM> may select a first image frame and a second image frame from among the plurality of image frames obtained through the column-readout circuit <NUM> before the first time <NUM>. Based on the difference between the pixel value of the ROI in the selected first image frame and the pixel value of the ROI in the selected second image frame exceeding a preset threshold, the processor <NUM> may identify that the object in the ROI has moved between the time of obtaining the first image frame and the time of obtaining the second image frame. In this case, the controller <NUM> may set the time of obtaining the second image frame to the first time <NUM> and automatically start automatic slow motion recording from the first time <NUM>.

Meanwhile, the controller <NUM> may deliver, through the interface <NUM> to the external circuit <NUM>, a plurality of first image frames read out at 120fps from the column-readout circuit <NUM> before the first time <NUM>. For example, when the output speed of the interface <NUM> is 240fps, the controller <NUM> may deliver, in real-time, the plurality of first image frames, which are read out at 120fps, to the external circuit <NUM> without the need for storing the image frames in the memory <NUM>. Since the interface with an output speed of 240fps may deliver one image frame every <NUM>/<NUM> seconds, the controller <NUM> may deliver image frames used to provide a 120fps video once every <NUM>/<NUM> seconds (e.g., <NUM> sec, <NUM>/<NUM> sec, <NUM>/<NUM> sec,. ) while simultaneously delivering preview images once every <NUM>/<NUM> seconds (e.g., <NUM>/<NUM> sec, <NUM>/<NUM> sec, <NUM>/<NUM> sec,. The processor in the external circuit <NUM> may provide a 120fps video by using the plurality of first image frames output through the interface of the image sensor. The processor in the external circuit <NUM> may provide a preview by using the plurality of image frames output through the interface of the image sensor.

According to an embodiment, the controller <NUM> may raise the read-out speed of the column-readout circuit <NUM> to, e.g., 960fps, to perform automatic slow motion recording from the first time <NUM> when the object in the ROI moves. When the read-out speed rises from 120fps to 960fps, the number of image frames obtained increases from <NUM> per second to <NUM> per second, and thus, the movement of the object in the ROI may accurately be observed. The controller <NUM> may deliver, through the interface <NUM> to the external circuit <NUM>, a plurality of second image frames read out at 960fps from the column-readout circuit <NUM> right after the first time <NUM>. For example, when the output speed of the interface <NUM> is 960fps, the controller <NUM> may deliver, in real-time, the plurality of second image frames, which are read out at 960fps, to the external circuit <NUM> without the need for storing the image frames in the memory <NUM>.

Meanwhile, when the output speed of the interface <NUM> is lower than 960fps, the controller <NUM> may store the plurality of second image frames read out at 960fps in the memory <NUM> and may then deliver the plurality of second image frames stored in the memory to the external circuit <NUM> depending on the output speed of the interface <NUM>. For example, the controller <NUM> may store all of the plurality of second image frames read out at 960fps in the memory <NUM>. The controller <NUM> may deliver the plurality of second image frames stored through the interface <NUM> to the external circuit <NUM>. For example, when the output speed of the interface <NUM> is 240fps, the controller <NUM> may output a preview image at 120fps while simultaneously outputting, at 120fps, the plurality of second image frames stored in the memory <NUM>. The processor in the external circuit <NUM> may provide a 960fps video by using the plurality of second image frames output through the interface of the image sensor <NUM>. The controller <NUM> may store the plurality of second image frames read out at 960fps in the memory <NUM> while simultaneously outputting the plurality of second image frames stored in the memory <NUM> through the interface <NUM> at 120fps. Alternatively, the controller <NUM> may output the plurality of second image frames stored in the memory <NUM> through the interface <NUM> at 120fps from the moment that the storage of the memory <NUM> is full of the plurality of second image frames read out at 960fps.

According to an embodiment of the disclosure, based on the storage of the memory <NUM> being filled up based on a second time <NUM>, the controller <NUM> may pause automatic slow motion recording. The controller <NUM> may lower the read-out speed of the column-readout circuit <NUM> corresponding to pausing automatic slow motion recording. For example, the controller <NUM> may lower the read-out speed from 960fps to 120fps to deliver the read-out image frames, in real-time, to the external circuit <NUM> without the need for storing the image frames in the memory <NUM>. The column-readout circuit <NUM> may read-out a plurality of third image frames at 120fps from the second time <NUM>.

Meanwhile, the controller <NUM> may store the plurality of second image frames read out at 960fps from the first time <NUM> until the storage of the memory <NUM> is full, and the controller <NUM> may output the plurality of second image frames stored in the memory <NUM> from the second time <NUM> when the storage of the memory <NUM> is full. For example, when the output speed of the interface <NUM> is 240fps, the controller <NUM> may output the plurality of second image frames stored in the memory <NUM> at 120fps while simultaneously outputting, at 120fps, a plurality of third image frames read out in real-time. The processor in the external circuit <NUM> may provide a 960fps video by using the plurality of second image frames output through the interface of the image sensor. Further, the processor in the external circuit <NUM> may provide a 120fps video by using the plurality of third image frames output through the interface of the image sensor.

According to an embodiment of the disclosure, the controller <NUM> may obtain a signal related to starting manual slow motion recording from the user at a third time <NUM>. The controller <NUM> may raise the read-out speed of the column-readout circuit <NUM> to 960fpls corresponding to the obtained signal related to starting manual slow motion recording. The column-readout circuit <NUM> may read-out a plurality of fourth image frames at 960fps from the third time <NUM>. The processor in the external circuit <NUM> may provide a 960fps video by using the plurality of fourth image frames output through the interface of the image sensor.

According to an embodiment, based on the storage of the memory <NUM> being filled up at a fourth time <NUM>, the controller <NUM> may pause manual slow motion recording. The controller <NUM> may lower the read-out speed of the column-readout circuit <NUM> corresponding to pausing manual slow motion recording. For example, the controller <NUM> may lower the read-out speed from 960fps to 120fps to deliver the read-out image frames, in real-time, to the external circuit <NUM> without the need for storing the image frames in the memory <NUM>. The column-readout circuit <NUM> may read-out a plurality of fifth image frames at 120fps from the fourth time <NUM>. The processor in the external circuit <NUM> may provide a 120fps video by using the plurality of fifth image frames output through the interface of the image sensor.

According to an embodiment of the disclosure, the controller <NUM> included in the image sensor may set the read-out speed, which is the output speed of the column-readout circuit <NUM>, as 960fps corresponding to a request related to the execution of a slow motion recording mode. When the read-out speed is set to 960fps, the column-readout circuit <NUM> may read-out <NUM> image frames per second based on light obtained by the pixel array <NUM>.

Meanwhile, the controller <NUM> may select some of the plurality of image frames read out at 960fps in a <NUM>:<NUM> ratio. In other words, the controller <NUM> may store one out of every eight read-out image frames in the memory <NUM>. The controller <NUM> may deliver the image frames stored in the memory <NUM> to the external circuit <NUM>, and the processor in the external circuit <NUM> may provide a 120fps video by using the delivered image frames.

Or, the controller <NUM> may store all of the plurality of image frames read out at 960fps in the memory <NUM>. The controller <NUM> may deliver the image frames stored in the memory <NUM> to the external circuit <NUM>, and the processor <NUM> in the external circuit <NUM> may provide a 960fps video by using the delivered image frames.

In the above embodiments, the 960fps video obtained between the first time <NUM> and the second time <NUM> is described as an outcome of automatic slow motion recording, and the 960fps obtained between the third time <NUM> and the fourth time <NUM> is described as an outcome of the manual slow motion recording. However, this does not intend to limit the order of automatic slow motion recording and manual slow motion recording. Automatic slow motion recording and manual slow motion recording may selectively be performed, and they may be performed three or more times.

In the above embodiments, the controller <NUM> in the image sensor <NUM> has been described as the entity for all the operations. However, this is merely an example for one or more processors in the electronic device, and the operations based on the embodiments may also be performed by other various processors in the electronic device <NUM>, as well as by the controller <NUM>. For example, in various embodiments set forth herein, the controller <NUM> may be referred to as a first processor, and the processor <NUM> or <NUM> in the external circuit <NUM> may be referred to as a second processor. The frame rates, e.g., 120fps or 960fps, used in the above embodiments may vary depending on the settings of the electronic device or the performance of the interface.

<FIG> is a flowchart illustrating a method of recording a video in slow motion using an electronic device <NUM> according to an embodiment. The embodiment of <FIG> is described in detail with reference to <FIG>, <FIG>, and <FIG>. <FIG> is a view illustrating an example in which an electronic device <NUM> records a video in slow motion in response to the movement of an object in an ROI <NUM>, according to an embodiment. <FIG> is a view illustrating an example of detecting the movement of an electronic device by using at least one background region <NUM>, <NUM>, <NUM>, and <NUM> or ROI <NUM> in an image according to an embodiment. <FIG> is a graph illustrating a variation, over time, in pixel value in an ROI corresponding to the movement of an electronic device <NUM> according to an embodiment.

In operation <NUM>, an electronic device <NUM> (in particular the processor <NUM>) obtains a plurality of first images including an ROI based on a first frame rate by using the camera module <NUM>. For example, in response to receiving an input regarding video recording from the user, the electronic device <NUM> obtains the plurality of first images based on the first frame rate (e.g., 30fps or 60fps). The plurality of first images includes an ROI. The electronic device <NUM> may set a range of interest of an object or subject to identify to initiate slow motion recording through the ROI.

According to an embodiment, the electronic device <NUM> may identify the movement of the electronic device <NUM> during a predetermined time by using the sensor module <NUM>. According to an embodiment, the electronic device <NUM> may identify sensor data through the sensor module <NUM> during the predetermined time, and based on the sensor data being less than a threshold, the electronic device <NUM> may identify that there is no movement of the electronic device <NUM>. According to an embodiment, based on there being no movement of the electronic device <NUM> during the predetermined time, the electronic device <NUM> may operate in slow motion recording standby mode. According to an embodiment, the slow motion recording standby mode may mean a state to initiate slow motion recording, as a state to identify the movement of an object in the ROI <NUM>.

According to an embodiment, the electronic device <NUM> may identify the reception of a user input related to the slow motion recording standby mode. For example, as shown in <<NUM>> and <<NUM>> of <FIG>, the electronic device <NUM> may operate in the slow motion recording standby mode in response to the reception of a user input through a slow motion recording-related user interface <NUM>.

According to an embodiment, the electronic device <NUM> may abstain from displaying, or may deactivate, the ROI <NUM> until before operating in the slow motion recording standby mode. For example, as shown in <<NUM>> of <FIG>, the electronic device <NUM> may abstain from displaying, or may deactivate, the ROI <NUM> until before operating in the slow motion recording standby mode. According to an embodiment, in response to operating in the slow motion recording standby mode, the electronic device <NUM> may display or activate the ROI <NUM>. For example, as shown in <<NUM>> of <FIG>, in response to operating in the slow motion recording standby mode, the electronic device <NUM> may display or activate the ROI <NUM>.

In operation <NUM>, according to an embodiment, the electronic device <NUM> may detect the movement of the electronic device <NUM> by using the sensor module <NUM> in the slow motion recording standby mode. According to an embodiment, the electronic device <NUM> may detect the movement of the electronic device <NUM> based on at least one of a gyro sensor or an acceleration sensor. As a sensor to detect the movement of the electronic device <NUM>, the gyro sensor or the acceleration sensor is merely an example, but is not limited thereto.

According to an embodiment, the electronic device <NUM> may detect the movement of the electronic device <NUM> based on at least one of the movement of an object in the background regions <NUM>, <NUM>, <NUM>, and <NUM> of the first images or the movement of an object in the ROI <NUM>. For example, as shown in <FIG>, the plurality of first images may include at least one background region <NUM>, <NUM>, <NUM>, and <NUM> or the ROI <NUM>. The electronic device <NUM> may identify the movement of an object in the background region <NUM> based on a pixel variation in the background region and may identify the movement of an object in the ROI <NUM> based on the pixel variation in the ROI <NUM>.

According to an embodiment, the electronic device <NUM> may set at least one background region <NUM>, <NUM>, <NUM>, and <NUM> based on at least some of image regions displayed on the display device <NUM> or set a background region in an external region (e.g., the margin region of the image sensor <NUM>) other than the image regions not displayed on the display device <NUM>. According to an embodiment, the background regions <NUM>, <NUM>, <NUM>, and <NUM> may be set as ones away by a preset distance from the ROI <NUM> set by the user or as ones including the object corresponding to the focal distance of the image sensor <NUM>. According to an embodiment, the background regions <NUM>, <NUM>, <NUM>, and <NUM> may be set as ones including objects positioned in the same distance as the distance between the electronic device <NUM> and the object in the ROI <NUM> or positioned within a predetermined distance range. According to an embodiment, the electronic device <NUM> may identify the pixel variation in the background regions whose contrast value exceeds a preset value from among the background regions <NUM>, <NUM>, <NUM>, and <NUM> while excluding the background regions whose contrast value is not more than the preset value from among the background regions <NUM>, <NUM>, <NUM>, and <NUM>.

In operation <NUM>, based on the movement of the electronic device <NUM> meeting a first predetermined range, the electronic device <NUM> identifies a threshold related to the movement of the ROI as a first threshold.

The electronic device <NUM> identifies whether the movement of the electronic device <NUM> meets the first predetermined range. The electronic device <NUM> either identifies whether a sensor data value obtained through the sensor module <NUM> meets the first predetermined value.

Based on the movement of the electronic device <NUM> meeting the first predetermined range, the electronic device <NUM> identifies the first threshold based on the pixel variation identified while the electronic device <NUM> moves. For example, referring to <FIG>, the electronic device <NUM> may display a preview image <NUM> for the object being captured on the display device <NUM> from time t0 or from before t0. In this case, the electronic device <NUM> may obtain a plurality of first images read out (<NUM>) at 60fps through the image sensor <NUM> and provide a 60fps video <NUM> by using at least some of the plurality of first images. During video recording, the electronic device <NUM> may store the plurality of read-out first images in the memory <NUM>, and during a time period from t1 to t2, the electronic device <NUM> may enter a mode for slow motion video recording based on the moving state of the electronic device <NUM> or by receiving a touch input for slow motion video recording from the user through a user interface. After t2, the electronic device <NUM> may change the read-out speed of the image sensor <NUM> from a first speed (60fps) to a second speed (960fps) and obtain a plurality of images. The time period from t2 to t3 relates to a standby mode in which the electronic device <NUM> records video in slow motion. The electronic device <NUM> may obtain a plurality of images read out (<NUM>) at 960fps and provide a 60fps video <NUM> by using at least some of the plurality of images. The electronic device <NUM> may identify a pixel variation in the ROI <NUM> due to the movement of the electronic device <NUM> and identify the maximum value <NUM>, the mean value <NUM>, or the second maximum value of the pixel variation. The electronic device <NUM> may identify a threshold that is a pixel value less than a preset value to exclude pixel values due to a sensor malfunction. The electronic device <NUM> may apply a weight <NUM> to one corresponding value (e.g., the mean value <NUM> or the maximum value <NUM>) of the pixel variation and identify a first threshold <NUM>. According to an embodiment, the electronic device <NUM> may identify a pixel variation in each ROI <NUM> or identify a variation in the mean value of all the pixels of the ROI <NUM>, but embodiments of the disclosure are not limited thereto. According to an embodiment, the electronic device <NUM> may identify the first threshold <NUM> by using at least one of the processor <NUM>, the image signal processor <NUM> in the camera module <NUM>, or the controller <NUM> in the image sensor <NUM>.

According to an embodiment, based on the reception of a signal related to video recording being a user's input received through the user interface while obtaining the plurality of first images, the electronic device <NUM> may identify, as the first threshold, a threshold related to the movement of the ROI based on the movement of the electronic device <NUM> measured after a preset period since the signal is obtained. For example, as shown in <FIG>, the electronic device <NUM> may receive a touch input for slow motion video recording from the user at t1, and to disregard the movement of the electronic device <NUM> while the touch input occurs, the electronic device <NUM> may identify, as the first threshold, a threshold related to the movement of the ROI based on the movement of the electronic device <NUM> measured from t2 which is a time after a preset period since t1 when the touch input occurs. The video recording-related signal may include, but is not limited to, e.g., a signal for entering the super slow motion video recording standby mode based on the movement of the electronic device <NUM> or a user input for initiating video recording or super slow motion video recording.

According to an embodiment, the electronic device <NUM> may identify a threshold related to the movement of the ROI as the identified first threshold. For example, the electronic device <NUM> may identify a threshold related to an object moving in the ROI <NUM> as the identified first threshold. According to an embodiment, the threshold related to the movement of the ROI <NUM> may mean a reference value for slow motion video recording, based on the pixel variation due to the movement of the object in the ROI <NUM> being not less than the threshold. For example, as shown in <FIG>, the electronic device <NUM> may identify the threshold related to the movement of the ROI <NUM> at t3 as the identified first threshold and identify whether the pixel variation due to the movement of the object in the ROI <NUM> exceeds the first threshold <NUM>.

In operation <NUM>, based on the movement of the electronic device <NUM> meeting a second predetermined range larger than the first predetermined range, the electronic device <NUM> identifies a threshold related to the movement of the ROI as a second threshold <NUM> larger than the first threshold <NUM>.

According to an embodiment, the electronic device <NUM> may identify whether the movement of the electronic device <NUM> meets the second predetermined range. For example, the electronic device <NUM> may identify whether a sensor data value obtained through the sensor module <NUM> meets the second predetermined range larger than the first predetermined value. As another example, the electronic device <NUM> may identify a pixel variation corresponding to the movement of an object in the background regions in the plurality of first images or a pixel variation corresponding to the movement of an object in the ROI and identify whether the pixel variation meets the second predetermined range larger than the first predetermined range.

According to an embodiment, based on the movement of the electronic device <NUM> meeting the second predetermined range larger than the first predetermined range, the electronic device <NUM> may identify the second threshold <NUM> based on the pixel variation identified while the electronic device <NUM> moves. The method of identifying the second threshold <NUM> may adopt the method of identifying the first threshold <NUM> described above in connection with operation <NUM>. According to an embodiment, the electronic device <NUM> may identify the second threshold <NUM> by using at least one of the processor <NUM>, the image signal processor <NUM> in the camera module <NUM>, or the controller <NUM> in the image sensor <NUM>. According to an embodiment, the electronic device <NUM> may identify the threshold related to the movement of the ROI as the second threshold <NUM> based on the movement of the electronic device measured, after a preset period since a video recording-related signal while obtaining the plurality of first images. For example, referring to <FIG>, the electronic device <NUM> may identify the second threshold <NUM> based on the movement of the electronic device <NUM> measured, from t2 which is a time after a preset period since t1 when a touch input occurs and may identify a threshold related to the movement of the ROI as the identified second threshold <NUM>.

According to an embodiment, the electronic device <NUM> may identify the illuminance of the outside of the electronic device <NUM> by using the sensor module <NUM> (e.g., an illuminance sensor). According to an embodiment, the electronic device <NUM> may identify illuminance information about the outside of the electronic device <NUM> by using exposure information or gain information identified through the image sensor <NUM>. According to an embodiment, the electronic device <NUM> may identify the threshold related to a movement of the ROI based on the movement of the electronic device and illuminance information about an outside of the electronic device obtained by using the sensor module <NUM>. For example, the electronic device <NUM> may identify a pixel variation in the ROI <NUM> based on the movement of the electronic device <NUM> and the ambient illuminance about the outside of the electronic device <NUM> during a preset time period (e.g., the time period from t2 to t3 as shown in <FIG>) and identify the maximum value or mean value of the pixel variation. The electronic device <NUM> may apply a weight to one value corresponding to the pixel variation, identifying the threshold. Based on the illuminance information meeting a third predetermined range, the electronic device <NUM> may identify the threshold related to the movement of the ROI <NUM> as a third threshold. Based on the illuminance information meeting a fourth predetermined range smaller than the third predetermined range, the electronic device <NUM> may identify the threshold related to the movement of the ROI <NUM> as a fourth threshold larger or different than the third threshold. According to an embodiment, the electronic device <NUM> may identify the third threshold or the fourth threshold by using at least one of the processor <NUM>, the image signal processor <NUM> in the camera module <NUM>, or the controller <NUM> in the image sensor <NUM>.

According to an embodiment, the electronic device <NUM> may receive an input regarding the settings of the ROI <NUM> from the user through a user interface. For example, the electronic device <NUM> may receive an input regarding a position setting of the ROI <NUM> or an input regarding a size setting of the ROI <NUM> from the user. According to an embodiment, the electronic device <NUM> may identify a threshold related to the movement of the ROI as a fifth threshold based on the setting of the ROI <NUM>. For example, based on the size of a first ROI being <NUM>*<NUM> pixels, the electronic device <NUM> may identify the threshold related to the movement of the first ROI to the first ROI as A value. As another example, based on the size of the second ROI being <NUM>*<NUM> pixels smaller than the first ROI, the electronic device <NUM> may identify the threshold related to the movement of the second ROI to the second ROI as B value which is larger than the A value.

According to an embodiment, in operation <NUM>, the electronic device <NUM> may obtain a plurality of second images based on a second frame rate larger than the first frame rate, in response to an object moving at a corresponding one of the first threshold <NUM> and the second threshold <NUM> in the ROI set by the electronic device <NUM>. For example, as shown in <<NUM>> of <FIG>, the electronic device <NUM> may initiate slow motion recording to obtain the plurality of second images based on the second frame rate in response to the object <NUM> moving at a corresponding one of the first threshold <NUM> and the second threshold <NUM>. According to an embodiment, based on the pixel variation in the ROI <NUM> being larger than a corresponding one of the first threshold <NUM> and the second threshold <NUM>, the electronic device <NUM> may obtain the plurality of second images based on the second frame rate. For example, as shown in <FIG>, based on the threshold related to the movement of the ROI <NUM> being identified to the first threshold <NUM> or the second threshold <NUM> or based on the pixel variation due to the movement of the object in the ROI <NUM> being larger than the threshold related to the movement of the ROI <NUM> at t4, the electronic device <NUM> may obtain the plurality of second images (e.g., a video <NUM>) based on the second frame rate (e.g., 960fps). According to an embodiment, the electronic device <NUM> may identify the threshold related to a movement of the ROI <NUM> based on the movement of the electronic device and illuminance information about an outside of the electronic device obtained, and may obtain the plurality of second images based on the second frame rate larger than the first frame rate in response to the object moving at the threshold, identified based on the movement of the electronic device and the illuminance information about an outside of the electronic device, in the ROI <NUM>. According to an embodiment, the electronic device <NUM> may identify the threshold related to the movement of the ROI <NUM> as the third threshold or the fourth threshold larger than the third threshold based on the identified illuminance information about the outside of the electronic device <NUM> and may obtain the plurality of second images based on the second frame rate larger than the first frame rate in response to the object moving at the third threshold or the fourth threshold in the ROI <NUM>. For example, based on the pixel variation in the ROI <NUM> being larger than a corresponding one of the third threshold and the fourth threshold, the electronic device <NUM> may obtain the plurality of second images based on the second frame rate.

According to an embodiment, the electronic device <NUM> may identify the threshold related to the movement of the ROI <NUM> as the fifth threshold based on the size setting of the ROI <NUM> and obtain the second images based on the second frame rate in response to the object moving at the fifth threshold identified based on the setting of the ROI <NUM> in the ROI <NUM>. For example, based on the pixel variation in the ROI <NUM> being larger than the fifth threshold, the electronic device <NUM> may obtain the plurality of second images based on the second frame rate.

According to an embodiment, based on the electronic device <NUM> obtaining the plurality of second images based on the second frame rate, the electronic device <NUM> may alter the interface related to the ROI <NUM> or the user interface <NUM> related to slow motion recording. For example, as shown in <<NUM>> of <FIG>, the electronic device <NUM> may gradually enlarge or shrink the user interface displaying the ROI <NUM> while obtaining the plurality of second images based on the second frame rate. As another example, as shown in <<NUM>> of <FIG>, the electronic device <NUM> may indicate that the slow motion recording-related user interface <NUM> is being activated while obtaining the plurality of second images based on the second frame rate.

According to an embodiment, when a predetermined time elapses from the time when the plurality of second images begin obtaining, the electronic device <NUM> may pause the acquisition of the plurality of second images and obtain the plurality of first images at the first frame rate. For example, as shown in <<NUM>> of <FIG>, when a predetermined time elapses since the slow motion recording begins, the electronic device <NUM> may terminate the slow motion recording by pausing the acquisition of the plurality of second images. According to an embodiment, when the electronic device <NUM> stores as many second images as a predetermined number of image frames or a predetermined capacity, the electronic device <NUM> may pause the acquisition of the plurality of second images and obtain the plurality of first images at the first frame rate. According to an embodiment, in response to identifying the reception of a user input through a user interface (not shown) related to the termination of slow motion recording, the electronic device <NUM> may pause the acquisition of the plurality of second images and obtain the plurality of first images at the first frame rate.

According to an embodiment, in response to pausing the acquisition of the plurality of second images, the electronic device <NUM> may alter the slow motion recording-related user interface <NUM>. For example, based on the acquisition of the plurality of second images pausing as shown in <<NUM>> of <FIG>, the electronic device <NUM> may indicate that the slow motion recording-related user interface <NUM> is being deactivated. According to an embodiment, in operation <NUM>, the electronic device <NUM> may provide a video related to the moving object by using at least some of the plurality of first images and at least some of the plurality of second images. For example, as shown in <FIG>, the electronic device <NUM> may provide a 60fps video <NUM> during a time period from t0 to t4 and a 960fps video after t4. According to an embodiment, the electronic device <NUM> may perform operations <NUM> to <NUM> by using at least one of the processor <NUM>, the image signal processor <NUM> in the camera module <NUM>, or the controller <NUM> in the image sensor <NUM>. According to an embodiment, the electronic device <NUM> may provide the video related to the moving object using at least some of the plurality of first images and at least some of the plurality of second images obtained in response to the object moving at the threshold identified based on the movement of the electronic device and the illuminance information about an outside of the electronic device.

<FIG> is a table representing weights to identify a threshold related to an object moving in an ROI <NUM> based on at least of the movement or illuminance of an electronic device <NUM> or the size of the ROI <NUM> according to an embodiment.

According to an embodiment, it is assumed that the electronic device <NUM> remains the same condition regarding the external illuminance of the electronic device <NUM> and the size of the ROI <NUM> but is subject to differences in movement. According to an embodiment, based on the outside of the electronic device <NUM> being a high illuminance, the size of the ROI <NUM> being <NUM>*<NUM> pixels, and the electronic device <NUM> being held in the user's hand so that the movement of the electronic device <NUM> due to the user's handshaking can be detected, the electronic device <NUM> may apply a preset first weight <NUM> and a preset second weight <NUM> to one corresponding value (e.g., the maximum value) among the pixel values in the ROI <NUM> which is varied by at least one of the movement of the electronic device <NUM>, ambient illuminance, or the size of the ROI <NUM>, thereby identifying a threshold. For example, the electronic device <NUM> may multiply one corresponding value (e.g., the maximum value) among the pixel values in the ROI <NUM> by the preset second weight <NUM> (Coeff), e.g., <NUM>, and then add the preset first weight <NUM> (Offset), e.g., <NUM>. According to an embodiment, based on the outside of the electronic device <NUM> including a high illuminance, the size of the ROI <NUM> being <NUM>*<NUM> pixels, and the electronic device <NUM> being mounted on a tripod, the electronic device <NUM> may apply a preset third weight <NUM> and a preset fourth weight <NUM> to one corresponding value (e.g., the maximum value) among the pixel values in the ROI <NUM> which is varied by at least one of the movement of the electronic device <NUM>, ambient illuminance, or the size of the ROI <NUM>, thereby identifying a threshold. For example, the electronic device <NUM> may multiply one corresponding value (e.g., the maximum value) among the pixel values in the ROI <NUM> by the preset fourth weight <NUM> (Coeff), e.g., <NUM>, and then add the preset third weight <NUM> (Offset), e.g., <NUM>.

According to an embodiment, it is assumed that the electronic device <NUM> remains the same condition regarding the external illuminance of the electronic device 101and the movement of the electronic device <NUM> but is subject to differences in the size of the ROI <NUM>. According to an embodiment, based on the outside of the electronic device <NUM> being a relatively high illuminance, the electronic device <NUM> being mounted on a tripod, and the size of the ROI <NUM> being <NUM>*<NUM> pixels, the electronic device <NUM> may apply the preset third weight <NUM> and the preset fourth weight <NUM> to one corresponding value (e.g., the maximum value) among the pixel values in the ROI <NUM> varied by at least one of the movement of the electronic device <NUM>, ambient illuminance, or the size of the ROI <NUM>, thereby identifying a threshold. For example, the electronic device <NUM> may multiply one corresponding value (e.g., the maximum value) among the pixel values in the ROI <NUM> by the preset fourth weight <NUM> (Coeff), e.g., <NUM>, and then add the preset third weight <NUM> (Offset), e.g., <NUM>. According to an embodiment, based on the outside of the electronic device <NUM> being a high illuminance, the electronic device <NUM> being mounted on a tripod, and the size of the ROI <NUM> being <NUM>*<NUM> pixels, the electronic device <NUM> may apply a preset fifth weight <NUM> and a preset sixth weight <NUM> to one corresponding value (e.g., the maximum value) among the pixel values in the ROI <NUM> varied by at least one of the movement of the electronic device <NUM>, ambient illuminance, or the size of the ROI <NUM>, thereby identifying a threshold. For example, the electronic device <NUM> may multiply one corresponding value (e.g., the maximum value) among the pixel values in the ROI <NUM> by the preset sixth weight <NUM> (Coeff), e.g., <NUM>, and then add the preset fifth weight <NUM> (Offset), e.g., <NUM>.

Although a threshold is identified by multiplying weight by a predetermined coefficient and then adding an offset above, it is appreciated by a skilled artisan that this is merely an example and identifying a threshold is not limited thereto.

<FIG> is a flowchart illustrating a method for recording a video in slow motion using an ROI <NUM> and background regions <NUM>, <NUM>, <NUM>, and <NUM> of an electronic device <NUM> according to an embodiment. The embodiment related to <FIG> is described in greater detail with reference to <FIG> is a view illustrating an example of recording a video in slow motion using an ROI <NUM> and at least one background region <NUM>, <NUM>, <NUM>, and <NUM> in an image according to an embodiment.

In operation <NUM>, an electronic device <NUM> (in particular the processor <NUM>) obtains a plurality of first images including an ROI <NUM> and possibly including at least one background region <NUM>, <NUM>, <NUM>, and <NUM> at a first frame rate by using the camera <NUM>. For example, the electronic device <NUM> may obtain images read out at 60fps in response to a video recording start signal and store the images, as a video, in the memory <NUM>.

According to an embodiment, in operation <NUM>, the electronic device <NUM> may obtain a first pixel variation in the ROI <NUM> in each of the plurality of first images. According to an embodiment, the electronic device <NUM> may obtain pixel variations in the ROI <NUM> in the Nth image frame and the ROI <NUM> in the N+1th image frame.

According to an embodiment, in operation <NUM>, the electronic device <NUM> may obtain a second pixel variation in at least one background region <NUM>, <NUM>, <NUM>, and <NUM> in each of the plurality of first images. According to an embodiment, the electronic device <NUM> may obtain pixel variations in at least one background region <NUM>, <NUM>, <NUM>, and <NUM> in the Nth image frame and at least one background region <NUM>, <NUM>, <NUM>, and <NUM> in the N+1th image frame.

According to an embodiment, the electronic device <NUM> may receive an input regarding the settings of the ROI <NUM> from the user through a user interface, and based on at least a portion of the ROI <NUM> overlapping at least a portion of the at least one background region, identify the second pixel variation by using the rest of the background region other than the overlapping portion. For example, the electronic device <NUM> may receive an input regarding the settings of the position or size of the ROI <NUM> from the user through the user interface. Based on at least a portion of the ROI <NUM> overlapping at least a portion of the background region <NUM>, <NUM>, <NUM>, and <NUM>, the electronic device <NUM> may identify the second pixel variation by using the rest of the background region other than the overlapping portion to prevent a malfunction due to the movement of the object in the overlapping portion.

In operation <NUM>, the electronic device <NUM> identifies whether the first pixel variation exceeds the first threshold. For example, as shown in <FIG>, the electronic device <NUM> may identify whether the first pixel variation due to the movement of the object in the ROI <NUM> exceeds the first threshold which is a threshold related to the object moving in the ROI <NUM>.

According to an embodiment, in operation <NUM>, the electronic device <NUM> may identify whether the second pixel variation is less than the second threshold. To prevent it from being identified that the object in the ROI <NUM> is moved as the electronic device <NUM> shakes, the electronic device <NUM> may identify whether the pixel variation in the ROI <NUM> comes from the electronic device <NUM> shaking or the movement of the object by using the background regions <NUM>, <NUM>, <NUM>, and <NUM>. For example, as shown in <FIG>, the electronic device <NUM> may identify whether the second pixel variation due to the movement of the object in the background regions <NUM>, <NUM>, <NUM>, and <NUM> corresponding to the movement of the electronic device <NUM> is less than the second threshold which is a threshold related to the object moving in the background regions <NUM>, <NUM>, <NUM>, and <NUM>. According to an embodiment, the second threshold may be the same or different from the first threshold.

In operation <NUM>, in response to identifying the first pixel variation exceeding the first threshold and the second pixel variation less than the second threshold, the electronic device <NUM> obtains a plurality of second images based on the second frame rate larger than the first frame rate. For example, as shown in <FIG>, based on the object in the ROI <NUM> moving at a preset speed or faster, and the object in the background regions <NUM>, <NUM>, <NUM>, and <NUM> moving slower than the preset speed, the electronic device <NUM> may obtain the plurality of second images based on the second frame rate (e.g., 960fps) larger than the first frame rate (e.g., 60fps). According to an embodiment, based on the difference between the first pixel variation in the ROI <NUM> and the second pixel variation in the background regions <NUM>, <NUM>, <NUM>, and <NUM> being not less than a first predetermined value, the electronic device <NUM> may identify the plurality of second images based on the second frame rate larger than the first frame rate.

In operation <NUM>, the electronic device <NUM> provides a video related to the moving object by using at least some of the plurality of first images and at least some of the plurality of second images. The method of providing the video may adopt the operations of <FIG> and operation <NUM> of <FIG>.

<FIG> is a flowchart illustrating a method for recording a video in slow motion using an ROI <NUM> in an image and a sensor module <NUM> of an electronic device <NUM> according to an embodiment.

In operation <NUM>, an electronic device <NUM> (in particular the processor <NUM>) obtains a plurality of first images including an ROI <NUM> based on a first frame rate. For example, the electronic device <NUM> may obtain images read out at 60fps in response to a video recording start signal and store the images, as a video, in the memory <NUM>.

In operation <NUM>, the electronic device <NUM> identifies whether first movement information related to the movement of an object in the ROI <NUM> meets a first predetermined condition. According to an embodiment, the electronic device <NUM> may identify whether a pixel variation due to the movement of the object in the ROI <NUM> exceeds a threshold related to the object moving in the ROI <NUM>. Based on the pixel variation due to the movement of the object in the ROI <NUM> exceeding the threshold moving in the ROI <NUM>, the electronic device <NUM> may identify that the first movement information related to the movement of the object meets the first predetermined condition.

According to an embodiment, in operation <NUM>, the electronic device <NUM> may identify whether second movement information related to the movement of the electronic device <NUM> obtained through the sensor module <NUM> meets a second predetermined condition. According to an embodiment, the electronic device <NUM> may identify whether a variation in sensor data obtained from the sensor module <NUM> (e.g., an acceleration sensor or gyro sensor) is less than a preset range. Based on the variation in sensor data obtained from the sensor module <NUM> being less than the preset range, the electronic device <NUM> may identify that the second movement information related to the movement of the electronic device meets the second predetermined condition.

In operation <NUM>, based on the first movement information meeting the first predetermined condition, and the second movement information meeting the second predetermined condition, the electronic device <NUM> obtains the plurality of second images based on the second frame rate larger than the first frame rate. According to an embodiment, based on the object in the ROI <NUM> moving at a preset speed or faster, and the electronic device <NUM> moving slower than the preset speed, the electronic device <NUM> may obtain the plurality of second images based on the second frame rate (e.g., 960fps) larger than the first frame rate (e.g., 60fps).

The one or more instructions may include a code provided by a complier or a code executable by an interpreter.

The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded) online via an application store (e.g., Play StoreTM), or between two user devices (e.g., smart phones) directly. If distributed online, at least part of the computer program product may be temporarily provided or at least temporarily stored in the machine-readable storage medium, such as memory of the manufacturer's server, a server of the application store, or a relay server.

According to an embodiment, an electronic device <NUM> according to claim <NUM> is provided.

According to an embodiment, a method according to claim <NUM> is provided.

As is apparent from the foregoing description, according to various embodiments, an electronic device may perform super slow motion video recording under a condition appropriate for the state of the electronic device at the time of video recording given movement or ambient illuminance information about the electronic device and may obtain a super slow motion video in the user's desired quality. According to various embodiments, there may be provided a method for allowing an electronic device to begin super slow motion video recording at the user's desired time considering the movement of an object in an ROI in an image frame or the movement of an object outside the ROI.

While the disclosure has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure as defined by the appended claims.

Claim 1:
An electronic device (<NUM>), comprising:
a camera (<NUM>);
a sensor circuit (<NUM>);
a memory (<NUM>); and
a processor (<NUM>) configured to:
obtain a plurality of first images including a region of interest, ROI, (<NUM>) based on a first frame rate using the camera,
detect a movement of the electronic device using the sensor circuit (<NUM>),
identify a threshold related to the movement of the ROI as one of a first threshold based on the movement of the electronic device meeting a first predetermined range, or a second threshold that is larger than the first threshold based on the movement of the electronic device meeting a second predetermined range that is larger than the first predetermined range,
obtain a plurality of second images based on a second frame rate that is larger than the first frame rate based on a pixel variation in the ROI (<NUM>) exceeding the threshold, and
provide a video related to a moving object using at least some of the plurality of first images and at least some of the plurality of second images.