DEVICE FOR PROCESSING IMAGE AND OPERATING METHOD THEREOF

A processing device and an operating method of the processing device are provided. An embodiment of the present disclosure provides the processing device that obtains a reconstructed image of the raw image by compressing a raw image at a preset compression ratio, encoding the compressed raw image, and decoding the encoded image, recognizes, in the raw image, a target object and a first region of interest (ROI) including the target object and extracts a first feature vector from the target object recognized in the raw image, recognizes, in the reconstructed image, the target object and a second ROI including the target object and extracts a second feature vector from the target object recognized in the reconstructed image, determines whether the target object is recognized in the reconstructed image, based on a degree of overlap between the first and second ROIs and a degree of similarity between the first and second feature vectors, and adjusts the preset compression ratio according to whether the target object is recognized.

BACKGROUND

The present disclosure relates to a device for processing an image or a video and a method thereof. More particularly, the present disclosure relates to a device for performing image processing for recognizing a target object in the image or the video.

2. Description of Related Art

Recently, surveillance camera systems have been increasingly installed inside or outside buildings, on streets, etc., for various purposes, such as crime prevention, security, store management, etc. As an example of a surveillance camera system, IP camera systems are in widespread use. The IP camera systems are connected to each other through a wired or wireless network to transmit and receive image data.

In a surveillance camera system, it is desirable to accurately recognize a person's face from a video captured using a camera. In a general system using a device, due to constraints on storage capacity or network bandwidths, a video captured using a camera is compressed and encoded before transmission to an external server. In this case, when encoding is performed at an excessively high compression ratio, the quality of an image degrades so that a face, a hairstyle, clothing, etc. of a person to be recognized may not be identified in the image. Furthermore, in exceptional situations, such as limited network bandwidth, data packet loss due to network instability, an error in an encoding operation, data packet loss in a User Datagram Protocol (UDP) environment, etc., a decoded image frame degrades, and a person's face and the like cannot be recognized in the degraded image frame. When the person's face cannot be accurately recognized in the degraded image frame, the person is mistakenly recognized as another person, and thus, it is difficult to track an unauthorized person.

SUMMARY

The present disclosure relates to a method and a device for processing an image or a video captured using a camera, and more particularly, to a device and operating method thereof for determining a compression ratio capable of improving accuracy in recognition of a target object from the image or the video.

To solve the above technical problem, an embodiment of the present disclosure provides a method, performed by a device, of processing a video. According to an embodiment of the present disclosure, a method performed by a processing device, includes: obtaining a raw image from at least one imaging sensor; obtaining a reconstructed image of the raw image by compressing the raw image at a preset compression ratio, encoding the compressed raw image, and decoding the encoded image; recognizing, in the raw image, a target object and a first region of interest (ROI) including the target object, and extracting a first feature vector from the target object recognized in the raw image; recognizing, in the reconstructed image, the target object and a second ROI including the target object, and extracting a second feature vector from the target object recognized in the reconstructed image; determining whether the target object is recognized in the reconstructed image, based on a degree of overlap between the first ROI and the second ROI and a degree of similarity between the first feature vector and the second feature vector; and adjusting the preset compression ratio according to whether the target object is recognized.

According to an embodiment of the present disclosure, the determining of whether the target object is recognized comprises: calculating the degree of overlap between the first ROI and the second ROI; calculating the degree of similarity between the first feature vector and the second feature vector; comparing the calculated degree of overlap with a preset first threshold and comparing the calculated degree of similarity with a preset second threshold; and determining, based on results of the comparisons, whether the target object is recognized in the reconstructed image.

According to an embodiment of the present disclosure, the adjusting of the preset compression ratio based on the comparing between the calculated degree of overlap with the first threshold or the comparing between the calculated degree of similarity with the second threshold includes lowering a value of the preset compression ratio based on (i) the degree of overlap that is less than or equal to the first threshold or (ii) the degree of similarity that is less than or equal to the second threshold.

According to an embodiment of the present disclosure, the method further includes obtaining another reconstructed image of the raw image by compressing the raw image at an adjusted preset compression ratio, encoding the compressed raw image, and decoding the encoded image. The adjusted preset compression ratio has the value that is lowered based on (i) the degree of overlap that is less than or equal to the first threshold or (ii) the degree of similarity that is less than or equal to the second threshold.

According to an embodiment of the present disclosure, the adjusting of the preset compression ratio includes determining the preset compression ratio to obtain the reconstructed image as a final compression ratio based on the calculated degree of overlap exceeds the first threshold or the calculated degree of similarity exceeds the second threshold.

According to an embodiment of the present disclosure, the calculating of the degree of overlap includes: obtaining first position coordinate values of the first ROI that is a region where the target object is recognized from the raw image; obtaining second position coordinate values of the second ROI that is a region where the target object is recognized from the reconstructed image; and calculating the degree of overlap, which indicates a degree to which the first ROI and the second ROI overlap, by using the first position coordinate values and the second position coordinate values.

According to an embodiment of the present disclosure, the reconstructed image is obtained by compressing, encoding, and decoding the raw image by using at least one image compression standard, and the preset compression ratio is determined by a quantization parameter (QP) in the at least one image compression standard.

According to an embodiment of the present disclosure, a processing device includes: an imaging sensor configured to obtain a raw image by capturing an image of an object; an encoder including an imaging sensor configured to obtain a raw image by capturing an image of an object; an encoder including an encoding module configured to generate encoded image data by compressing the raw image at a preset compression ratio and encoding the compressed raw image into image data and a decoding module configured to generate a reconstructed image of the raw image by decoding the encoded image data; a network interface configured to receive the raw image and the reconstructed image; a memory storing a program including one or more instructions; and a processor configured to execute the one or more instructions of the program stored in the memory to: recognize, in the raw image, a target object and a first region of interest (ROI) including the target object, and extract a first feature vector from the target object recognized in the raw image, recognize, in the reconstructed image, the target object and a second ROI including the target object, and extract a second feature vector from the target object recognized in the reconstructed image, determine whether the target object is recognized in the reconstructed image, based on a degree of overlap between the first ROI and the second ROI and a degree of similarity between the first feature vector and the second feature vector, and adjust the preset compression ratio according to whether the target object is recognized.

According to an embodiment of the present disclosure, the processor is further configured to calculate the degree of overlap between the first ROI and the second ROI, calculate the degree of similarity between the first feature vector and the second feature vector, compare the calculated degree of overlap with a preset first threshold and compare the calculated degree of similarity with a preset second threshold, and determine, based on results of the comparisons, whether the target object is recognized in the reconstructed image.

According to an embodiment of the present disclosure, the processor is further configured to lower a value of the preset compression ratio based on (i) the degree of overlap that is less than or equal to the first threshold or (ii) the degree of similarity that is less than or equal to the second threshold.

According to an embodiment of the present disclosure, the processor is further configured to obtain another reconstructed image of the raw image by compressing the raw image at an adjusted preset compression ratio, encoding the compressed raw image, and decoding the encoded image. The adjusted preset compression ratio has the value that is lowered based on (i) the degree of overlap that is less than or equal to the first threshold or (ii) the degree of similarity that is less than or equal to the second threshold.

According to an embodiment of the present disclosure, the processor is further configured to determine to obtain the reconstructed image as a final compression ratio based on the calculated degree of overlap exceeds the first threshold or the calculated degree of similarity exceeds the second threshold.

According to an embodiment of the present disclosure, the processor is further configured to: obtain first position coordinate values of the first ROI that is a region where the target object is recognized from the raw image, obtain second position coordinate values of the second ROI that is a region where the target object is recognized from the reconstructed image, and calculate the degree of overlap, which indicates a degree to which the first ROI and the second ROI overlap, by using the first position coordinate values and the second position coordinate values.

According to an embodiment of the present disclosure, the reconstructed image is obtained by compressing, encoding, and decoding the raw image by using at least one image compression standard, and the preset compression ratio is determined by a quantization parameter (QP) in the at least one image compression standard.

According to an embodiment of the present disclosure, at least one image compression standard is at least one of H.264/advanced video coding (AVC) or high efficiency video coding (HEVC) (H.265).

According to an embodiment of the present disclosure, the at least one image compression standard is at least one of H.264/advanced video coding (AVC) or high efficiency video coding (HEVC) (H.265).

To solve the above technical problem, another embodiment of the present disclosure provides a computer-readable recording medium having recorded thereon a program executed by a computer. According to an embodiment of the present disclosure, a computer program product includes a computer-readable storage medium. The computer-readable storage medium includes instructions performed by a processing device, the instructions for: obtaining a raw image from at least one imaging sensor; obtaining a reconstructed image of the raw image by compressing the raw image at a preset compression ratio, encoding the compressed raw image, and decoding the encoded image; recognizing, in the raw image, a target object and a first region of interest (ROI) including the target object, and extracting a first feature vector from the target object recognized in the raw image; recognizing, in the reconstructed image, the target object and a second ROI including the target object, and extracting a second feature vector from the target object recognized in the reconstructed image; calculating a degree of overlap between the first ROI and the second ROI and calculating a degree of similarity between the first feature vector and the second feature vector; comparing the calculated degree of overlap with a first threshold and comparing the calculated degree of similarity with a second threshold; and adjusting the preset compression ratio according to whether the target object is recognized.

DETAILED DESCRIPTION

As the terms used herein, general terms that are currently widely used are selected by taking functions in the present disclosure into account, but the terms may have different meanings according to an intention of one of ordinary skill in the art, precedent cases, advent of new technologies, etc. Furthermore, specific terms may be arbitrarily selected by the applicant, and in this case, the meaning of the selected terms will be described in detail in the detailed description of a corresponding embodiment. Thus, the terms used herein should be defined not by simple appellations thereof but based on the meaning of the terms together with the overall description of the present disclosure.

Singular expressions used herein are intended to include plural expressions as well unless the context clearly indicates otherwise. All the terms used herein, which include technical or scientific terms, may have the same meaning that is generally understood by a person of ordinary skill in the art.

Throughout the present disclosure, when a part “includes” or “comprises” an element, unless there is a particular description contrary thereto, the part may further include other elements, not excluding the other elements. Furthermore, terms, such as “portion,” “module,” etc., used herein indicate a unit for processing at least one function or operation and may be embodied as hardware or software or a combination of hardware and software.

The expression “configured to (or set to)” used herein may be used interchangeably, according to context, with the expression such as “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of”. The term “configured to (or set to)” may not necessarily mean only “specifically designed to” in terms of hardware. Instead, the expression “a system configured to” may mean, in some contexts, the system being “capable of”, together with other devices or components. For example, the expression “a processor configured to (or set to) perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) capable of performing the corresponding operations by executing one or more software programs stored in a memory.

In the present disclosure, a ‘device’ is an electronic device including at least one of an IP camera system and a mobile device.

In the present disclosure, an ‘image’ refers to a moving image or a video including at least one frame.

In the present disclosure, a ‘raw image’ is uncompressed image data obtained as an image captured using a camera is not compressed.

In the present disclosure, a ‘compression ratio’ represents an image compression parameter for compressing a raw image by using a method that matches a standard specification. Compression ratios may exist in various forms for each standard specification. For example, in standards for playing a video (e.g., H.264/advanced video coding (AVC), high efficiency video coding (HEVC) (H.265), etc.), a compression ratio of an image may be determined based on a quantization parameter.

In the present disclosure, a ‘quantization parameter’ is a parameter indicating a degree of quantization of transform coefficients defined in a standard related to compression of a video, e.g., H.264/AVC or HEVC (H.265), and may be used to determine a compression ratio of an image. In an embodiment, a compression ratio is proportional to a value of a quantization parameter. For example, when the value of the quantization parameter is large, the compression ratio is high, and when the value of the quantization parameter is small, the compression ratio is low.

In the present disclosure, a ‘transform coefficient’ refers to a coefficient obtained by transforming residual data into a transform domain using various transform techniques, the residual data representing a difference value between a raw image and a predicted image frame (I-frame) generated via intra-prediction using a spatial correlation among a plurality of image frames or inter-prediction using a temporal correlation. As a method of transforming residual data into transform coefficients in H.264/AVC, a discrete cosine transform (DCT), a Hadamard transform, etc. may be used.

FIG.1Ais a block diagram illustrating a structure of an Internet Protocol (IP) camera system100according to an embodiment of the present disclosure.

In an embodiment, the IP camera system100is a surveillance camera system and may capture in real-time an image of a target area or a target object being monitored for the purpose of surveillance or security.

Referring toFIG.1A, the IP camera system100may include an IP camera1000and a processing device2000.

The IP camera1000may obtain a raw image by capturing an image of the target region or the target object being monitored and obtain a reconstructed image by compressing the raw image with a preset compression ratio, encoding the raw image into image data, and decoding the encoded image data. The IP camera1000may be implemented as a pan-tilt-zoom (PTZ) camera or a fisheye camera, but is not limited thereto. Although only one IP camera1000is shown inFIG.1A, the present disclosure is not limited thereto. In an embodiment, the IP camera system100may include a plurality of IP cameras1000.

The IP camera1000transmits the raw image and the reconstructed image to the processing device2000(operation S100).

The IP camera1000may include an imaging sensor1100, an imaging processor1200, a memory1300, and a network interface1400.

The imaging sensor1100may obtain an image of a monitoring target area or target object captured through a lens. The image obtained via the imaging sensor1100is a raw image corresponding to uncompressed image data. The imaging sensor1100may include a photodiode array. The imaging sensor1100may include, for example, a charge-coupled device (CCD) module or a complementary metal-oxide-semiconductor (CMOS) module, but is not limited thereto.

Although one imaging sensor1100is shown inFIG.1A, the present disclosure is not limited thereto. In an embodiment, the imaging sensor1100may be configured as a plurality of imaging sensors.

The imaging processor1200may generate a reconstructed image by performing image processing on the raw image obtained by the imaging sensor1100. The imaging processor1200may compress the raw image at a preset compression ratio, encode the compressed raw image into image data, and decode the encoded image data by using an encoder1310stored in the memory1300, thereby generating a reconstructed image.

The imaging processor1200may include a hardware component having computational capabilities to perform known image processing tasks. For example, the imaging processor1200may include at least one of a CPU, a microprocessor, or a graphics processing unit (GPU). However, the imaging processor1200is not limited to the examples listed above.

The memory1300may include the encoder1310. The encoder1310is configured to generate a reconstructed image by compressing, encoding, and decoding a raw image using at least one of image compression standards, for example, H.264/AVC and HEVC (H.265).

The encoder1310may include an encoding module1320and a decoding module1330. The encoding module1320may be configured to encode an input image, i.e., a raw image, using a preset compression ratio and obtain encoded image data from the raw image. In an embodiment, by using the encoding module1320, the imaging processor1200may obtain prediction data from an input image, i.e., a raw image, via inter-prediction and intra-prediction, obtain residual data produced by subtracting the prediction data from the raw image, and obtain encoded image data by performing transformation and quantization on the residual data. For example, the encoded image data may be quantized transform coefficients.

The decoding module1330may be configured to generate a reconstructed image by decoding the encoded image data. In an embodiment, by using the decoding module1330, the imaging processor1200may reconstruct the residual data by performing inverse quantization and inverse transform on the encoded image data, i.e., the quantized transform coefficients, and obtain a reconstructed image by adding the prediction data obtained via inter-prediction and intra-prediction to the reconstructed residual data.

When using an image compression standard, such as H.264/ACV or HEVC (H.265), the encoder1310may include, for example, an inter-prediction module, an intra-prediction module, a deblocking module, a transform module, a quantization module, an inverse transform module, an inverse quantization module, and an entropy encoding module. In this case, the modules listed above perform inter-prediction and intra-prediction and encoding and decoding, and thus, are not completely divided into the encoding module1320and the decoding module1330, as shown inFIG.1A. The encoding module1320and the decoding module1330are an example for explaining transformation and flow of data shown inFIG.2.

When a reconstructed image is generated, for example, using H.264/AVC or HEVC (H.265), the reconstructed image may be obtained by compressing and encoding a raw image by applying different quantization parameter values according to a coding unit (CU) or a largest coding unit (LCU) and decoding encoded image data. In this case, a specific CU or LCU in which a target object (e.g., a person's face, hairstyle, clothing, posture, etc.) is detected among a plurality of CUs or LCUs in an image frame included in the raw image may be compressed by applying a lower quantization parameter value thereto than to other CUs or LCUs.

The network interface1400may perform data transmission and reception between the IP camera1000and the processing device2000. The network interface1400may perform data communication with the processing device2000by using at least one of wired and wireless communication methods including Ethernet, wired and wireless local area networks (LANs), Wi-Fi, Wi-Fi Direct (WFD), and Wireless Gigabit Alliance (WiGig). The IP camera1000may transmit obtained image data of the raw image and image data of the reconstructed image to the processing device2000via the network interface1400(operation S100).

The processing device2000is a computing device configured to obtain the raw image and the reconstructed image from the IP camera1000, determine whether a target object is accurately recognized in the raw image and the reconstructed image and determine a compression ratio for obtaining the reconstructed image based on a determining result. The processing device2000may be located in commercial facilities, such as stores and restaurants, or a home in which the IP camera system100is installed. The processing device2000may be, for example, a Personal Computer (PC), but is not limited thereto.

The processing device2000may include a network interface2100, a processor2200, and a memory2300.

The network interface2100may perform data transmission and reception between the processing device2000and the IP camera1000. The network interface2100may perform data communication with the IP camera1000by using at least one of the wired and wireless communication methods including Ethernet, wired and wireless LANs, Wi-Fi, WFD, and WiGig. In an embodiment, the network interface2100may receive the raw image and the reconstructed image from the IP camera1000.

The processor2200may execute one or more instructions of a program stored in the memory2300. The processor2200may be composed of hardware components for performing arithmetic, logic and input/output (I/O) operations and signal processing. For example, the processor2200may include at least one of a CPU, a microprocessor, a GPU, an AP, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs), but is not limited thereto.

The memory2300may include, for example, at least one of a flash memory-type memory, a hard disk-type memory, a multimedia card micro-type memory, a card-type memory (e.g., an SD card or an XD memory), random access memory (RAM), static RAM (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), PROM, magnetic memory, magnetic disks, and optical disks.

The memory2300may store a program including instructions related to execution of a function or operation of determining a compression ratio for encoding the raw image based on regions in which target objects are detected respectively in the raw image and the reconstructed image and feature vectors extracted from the target objects. The memory2300may store at least one of instructions, algorithms, data structures, and program code that are readable by the processor2200. The instructions, algorithms, data structures, and program code stored in the memory2300may be implemented by a programming or scripting language, such as C, C++, Java, assembler, etc.

In embodiments set forth below, the processor2200may be implemented by executing instructions or code of a program stored in the memory2300.

The memory2300may include an object recognition module2310, a feature vector extraction module2320, a recognition result determination module2330, and a compression ratio adjustment module2340. A plurality of ‘modules’ included in the memory2300indicate a unit for processing a function or operation performed by the processor2200, which may be implemented using software, such as instructions or program code.

The object recognition module2310is a module for recognizing a target object from a video. In an embodiment, the object recognition module2310may be configured to detect a target object in a video by using an artificial intelligence (AI) model including deep learning. In an embodiment, the object recognition module2310may detect a target object in a video by using a trained convolutional neural network (CNN) model. A target object refers to an object of interest that a user desires to detect in a video. The target object may include, for example, at least one of a person's face, hair, clothing, and posture.

The object recognition module2310may be configured to recognize the target object in the video by using model parameters trained via the CNN model. For example, when the target object is a face, the object recognition module2310may detect a person's face in a video by using a large scale dataset, such as CASIA-WebFace, VGGFaceNGGFace 2, or MS-Celeb-1M. In an embodiment, the object recognition module2310may include a MobileNet trained to perform face recognition using the VGGFace2 dataset.

In an embodiment, the processor2200may detect a target object in each of the raw image and the encoded image by using data and program code in the object recognition module2310. For example, the processor2200may obtain a person's face from each of the raw image and the reconstructed image by performing training using the model parameters trained via the CNN model included in the object recognition module2310. However, the present disclosure is not limited thereto, and the processor2200may detect at least one of a person's hairstyle, clothing, and posture in each of the raw image and the reconstructed image by performing training based on the trained model parameters included in the object recognition module2310.

However, a method, performed by the object recognition module2310, of detecting a target object is not limited to the above-described method. In an embodiment, the object recognition module2310may be configured to recognize a target object in a video by using an AI model including at least one of a recurrent neural network (RNN) model, support vector machine (SVM), linear regression, logistic regression, Naive Bayes classification, random forest, decision tree, and a k-nearest neighbor algorithm.

In an embodiment, the object recognition module2310may be configured to identify a bounding box of a target object detected in a video and obtain position coordinate values of the identified bounding box. By using data and program code in the object recognition module2310, the processor2200may identify a bounding box of a target object detected in the raw image and set a region included in the bounding box to be a first region of interest (ROI). By using the data and program code in the object recognition module2310, the processor2200may identify a bounding box of a target object detected in the reconstructed image and set a region included in the bounding box to be a second ROI.

The feature vector extraction module2320is a module for extracting a feature vector from an image of a detected target object. The feature vector extraction module2320may be configured to extract a feature vector from an image of a target object by using a known image processing method or an AI model including deep learning. For example, the feature vector extraction module2320may be configured to extract a feature vector from an image of a target object by using an independent component analysis (ICA) or principle component analysis (PCA) method.

For example, when an image of a target object shows a person's face, the feature vector extraction module2320may identify landmarks including the eyes, the nose, and the mouth (both mouth corners) of the face, obtain feature values (e.g., position coordinate values) corresponding to the identified landmarks, and extract a feature vector by using the obtained feature values. In another example, when an image of a target object shows a person's hairstyle, clothing, or specific posture, the feature vector extraction module2320may identify at least one key-point in the image of the target object, obtain feature values (e.g., position coordinate values) corresponding to the identified key-point, and extract a feature vector by using the obtained feature values. An embodiment in which a feature vector related to a person's hairstyle, clothing, or posture is extracted using the feature vector extraction module2320is described in detail below with reference toFIG.9.

In an embodiment, the feature vector extraction module2320may be configured to extract a feature vector from an image of a target object by using a CNN model. In an embodiment, the feature vector extraction module2320may extract feature values as a kernel having a preset size and a preset number of channels moves over an image of a target object with a stride. The feature vector extraction module2320may obtain a plurality of layers including the extracted feature values and obtain a feature vector map by applying weights to the plurality of layers. A rectified linear unit (ReLU) model may be used in the process of obtaining the feature vector value. In order to improve the efficiency of a CNN model, an operation of normalizing feature values through dropout and an operation of performing pooling or max pooling may be further added. The feature vector extraction module2320may be configured to extract a feature vector by using a CNN model, such as MobileNet v1/v2.

The processor2200may extract a feature vector from an image of a target object by using data and program code in the feature vector extraction module2320. In an embodiment, the processor2200may extract a first feature vector from an image of a target object recognized in the raw image and extract a second feature vector from an image of a target object recognized in the reconstructed image.

The recognition result determination module2330is a module for determining whether the target object is correctly recognized in the reconstructed image, based on a degree of overlap between ROIs detected respectively in the raw image and the reconstructed image and a degree of similarity between feature vectors respectively extracted from the raw image and the reconstructed image. The recognition result determination module2330may include an overlap degree calculation module2332, a similarity degree calculation module2334, and a comparison module2336.

The overlap degree calculation module2332is a module for calculating the degree of overlap among two or more images and outputting a resulting value. The overlap degree calculation module2332may obtain information about coordinate values of a bounding box on each of the two or more images. The overlap degree calculation module2332may be configured to calculate the degree of overlap by comparing the degree of overlap among two or more images using obtained coordinate values of corresponding bounding boxes.

In an embodiment, the overlap degree calculation module2332may calculate the degree of overlap among two or more images by using an Intersection Over Union (IoU) method for calculating the degree of overlap using coordinate value information. In the calculation of the degree of overlap using the IoU method, an IoU value of 1 means that two images perfectly or substantially overlap, and an IoU value of 0 means that the two images do not overlap at all. For example, when the IoU value is 0.5, it means that two-thirds of the two images overlap.

In an embodiment, the overlap degree calculation module2332may calculate the degree of overlap by calculating the sum of squares of errors between coordinate values of a plurality of images. For example, the overlap degree calculation module2332may calculate a coordinate value error by using coordinate values (x11, y11) of an upper-left corner of a first image, coordinate values (x12, y12) of a lower-right corner of the first image, coordinate values (x21, y21) of an upper-left corner of a second image, and coordinate values (x22, y22) of a lower-right corner of the second image, as shown in Equation 1 below.

However, the IoU method and the coordinate value error calculation method are examples of an overlap degree calculation method, and a method, performed by the overlap degree calculation module2332, of calculating the degree of overlap among two or more images is not limited to the IoU method and the coordinate value error calculation method.

By using data and program code in the overlap degree calculation module2332, the processor2200may calculate a degree of overlap between the first ROI including the target object recognized in the raw image and the second ROI including the target object recognized in the reconstructed image, based on coordinate value information of a bounding box of the first ROI and coordinate value information of a bounding box of the second ROI. In an embodiment, the processor2200may calculate the degree of overlap between the first ROI and the second ROI by using the IoU method or the coordinate value error calculation method provided by the overlap degree calculation module2332. In an embodiment, the processor2200may calculate a coordinate value error between the first and second ROIs by using coordinate values of an upper-left corner and a lower-right corner of a bounding box of the first ROI and coordinate values of an upper-left corner and a lower-right corner of a bounding box of the second ROI, and calculate the degree of overlap between the first and second ROIs based on the calculated coordinate value error.

The similarity degree calculation module2334is a module for calculating a degree of similarity between two feature vectors as a numerical value. In an embodiment, the similarity degree calculation module2334may be configured to calculate a correlation between two feature vectors as a numerical value.

The processor2200may calculate, as a numerical value, a degree of similarity between the first feature vector extracted from the target object detected in the raw image and the second feature vector extracted from the target object detected in the reconstructed image by using data and program code in the similarity degree calculation module2334. The processor2200may calculate the degree of similarity between the first feature vector and the second feature vector, for example, by using Equation 2 below.

In Equation 2 above, Forigand Fencrespectively denote a first feature vector extracted from a target object detected in the raw image and a second feature vector extracted from a target object detected in the reconstructed image, and N is a length of the feature vectors, μ is a mean value of the feature vectors, and σ is a standard deviation of the feature vectors.

Equation 2 is an example for calculating the degree of similarity, and a method, performed by the processor2200, of calculating the degree of similarity between the first and second feature vectors by using the similarity degree calculation module2334is not limited to Equation 2.

The comparison module2336is a module configured to compare the degree of overlap between the first and second ROIs, which is calculated using the overlap degree calculation module2332, with a first threshold, and compare the degree of similarity between the first and second feature vectors, which is calculated using the similarity degree calculation module2334, with a second threshold. For example, the first threshold may be 0.8, and the second threshold may be 0.5. However, numerical values of the first threshold and the second threshold are not limited to those described above. The processor2200may compare the degree of overlap between the first and second ROIs with the first threshold and compare the degree of similarity between the first and second feature vectors with the second threshold by using data and program code in the comparison module2336.

The comparison module2336may provide information about comparison results to the compression ratio adjustment module2340.

The compression ratio adjustment module2340is a module for determining a compression ratio at which the raw image is compressed to obtain the reconstructed image, based on whether the target object is recognized, which is determined by the recognition result determination module2330. Here, a ‘compression ratio’ represents an image compression parameter for encoding a raw image by using a method that matches a standard specification. Compression ratios may exist in various forms for each standard specification. For example, in standards for playing a video (e.g., H.264/AVC, HEVC (H.265), etc.), a compression ratio of an image may be determined based on a quantization parameter. In an embodiment, the compression ratio adjustment module2340may be configured to determine a value of a quantization parameter based on results of respectively comparing the degree of overlap and the degree of similarity obtained from the recognition result determination module2330with preset thresholds.

A ‘quantization parameter’ is a parameter indicating a degree of quantization of transform coefficients defined in a standard related to compression of a video, e.g., H.264/AVC or HEVC (H.265), and may be used to determine a compression ratio of an image. In an embodiment, a compression ratio is proportional to a value of a quantization parameter. For example, when the value of the quantization parameter is large, the compression ratio is high, and when the value of the quantization parameter is small, the compression ratio is low.

The compression ratio adjustment module2340may be configured to adjust a value of a compression ratio based on the comparison results provided by the comparison module2336. In an embodiment, by using data and program code in the compression ratio adjustment module2340, the processor2200may adjust a value of a quantization parameter, based on a result of comparing the degree of overlap between the first and second ROIs with the first threshold and a result of comparing the degree of similarity between the first and second feature vectors with the second threshold. The compression ratio adjustment module2340may change the value of the compression ratio by a preset unit.

In an embodiment, the processor2200may change the value of the compression ratio when the degree of overlap is less than or equal to the first threshold or the degree of similarity is less than or equal to the second threshold as a result of comparison using the comparison module2336. In this case, the processor2200may change the value of the compression ratio to a value lower than a compression ratio used to compress and encode the previously reconstructed image. In an embodiment, when the degree of overlap exceeds the first threshold and the degree of similarity exceeds the second threshold as a result of comparison using the comparison module2336, the processor2200may determine the compression ratio used to generate the reconstructed image as a final compression ratio.

The processing device2000transmits, to the IP camera1000, information about a compression ratio determined according to whether an object is recognized via the network interface2100(operation S110). In an embodiment, when the degree of overlap is less than or equal to the first threshold or the degree of similarity is less than or equal to the second threshold, the processor2200may transmit information about a changed compression ratio value to the IP camera1000via the network interface2100. In this case, the processor2200may repeat at least once operations of receiving, from the IP camera1000, via the network interface2100, image data of a reconstructed image generated through compression using the changed compression ratio, encoding, and decoding, recognizing a target object in the received reconstructed image and extracting a second feature vector from the target object, calculating the degree of overlap, calculating the degree of similarity, determining whether the target object is correctly recognized, and readjusting a compression ratio.

In an embodiment, when the degree of overlap exceeds the first threshold and the degree of similarity exceeds the second threshold, the processor2200may transmit information about a value of a final compression ratio to the IP camera1000via the network interface2100.

The IP camera1000may receive information about a compression ratio via the network interface1400and generate a bitstream by compressing the raw image using the received compression ratio and encoding the compressed raw image.

The IP camera1000may transmit, to an external server200, the bitstream generated by compressing the raw image at the determined compression ratio via the network interface1400(operation S120). In an embodiment, the IP camera1000may transmit the bitstream to the server200through a gateway. The gateway may connect the IP camera system100to the server200using a wired or wireless communication method. The server200may store the bitstream received from the IP camera1000in a storage210.

FIG.1Bis a block diagram illustrating a structure of a mobile device300according to an embodiment of the present disclosure.

The mobile device300may be an electronic device that processes a video obtained from a camera310and performs various calculations. For example, the mobile device300may be at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, or a desktop PC, a laptop PC, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, and a wearable device.

Referring toFIG.1B, the mobile device300may include a camera310, an encoder320, a processor330, a memory340, an image data storage350, and a communication interface360.

The camera310may include a lens312and an imaging sensor314. The camera310may image an object through the lens312and obtain an image of the object via the imaging sensor314. The image obtained via the imaging sensor314is a raw image corresponding to uncompressed image data. The imaging sensor314may include a photodiode array. The imaging sensor314may include, for example, a CCD module or a CMOS module, but is not limited thereto.

The encoder320is configured to generate a reconstructed image by compressing, encoding, and decoding a raw image using at least one of the image compression standards, for example, H.264/AVC and HEVC (H.265).

The encoder320may include an encoding module322and a decoding module324. The encoding module322may be configured to encode an input image, i.e., a raw image, using a preset compression ratio and obtain encoded image data from the raw image. In an embodiment, by using the encoding module322, the processor330may obtain prediction data from an input image, i.e., a raw image, via inter-prediction and intra-prediction, obtain residual data produced by subtracting the prediction data from the raw image, and obtain encoded image data by performing transformation and quantization on the residual data. For example, the encoded image data may be quantized transform coefficients.

The decoding module324may be configured to generate a reconstructed image by decoding the encoded image data. In an embodiment, by using the decoding module324, the processor330may reconstruct the residual data by performing inverse quantization and inverse transform on the encoded image data, i.e., the quantized transform coefficients, and obtain a reconstructed image by adding the prediction data obtained via inter-prediction and intra-prediction to the reconstructed residual data.

Because the encoding module322and the decoding module324shown inFIG.1Bperform the same functions and/or operations as the encoding module1320and the decoding module1330shown inFIG.1A, descriptions that are already provided above are omitted.

By executing at least one program code or instructions stored in the memory340, the processor330may determine whether a target object is accurately recognized in the raw image and the reconstructed image and determine a compression ratio for obtaining the reconstructed image based on a determining result. The processor330may be composed of hardware components for performing arithmetic, logic and input/output (I/O) operations and signal processing. In an embodiment, the processor330may be an AP configured to execute an application program developed for the mobile device300. However, the processor330is not limited thereto, and for example, it may include at least one of a CPU, a microprocessor, a GPU, ASICs, DSPs, DSPDs, PLDs, or FPGAs.

The memory340may include, for example, at least one of a flash memory-type memory, a hard disk-type memory, a multimedia card micro-type memory, a card-type memory (e.g., an SD card or an XD memory), RAM, SRAM, ROM, EEPROM, PROM, magnetic memory, magnetic disks, or optical disks.

The memory340may store a program including instructions related to execution of a function or operation of determining a compression ratio for encoding the raw image based on regions in which target objects are detected respectively in the raw image and the reconstructed image and feature vectors extracted from the target objects. The memory340may store at least one of instructions, algorithms, data structures, and program code that are readable by the processor330. The instructions, algorithms, data structures, and program code stored in the memory340may be implemented by a programming or scripting language, such as C, C++, Java, assembler, etc.

The memory340may include an object recognition module342, a feature vector extraction module344, a recognition result determination module346, and a compression ratio adjustment module348.

The object recognition module342is a module for recognizing a target object from a video. In an embodiment, the processor330may detect a target object in each of the raw image and the reconstructed image by using data and program code in the object recognition module342.

The feature vector extraction module344is a module for extracting a feature vector from an image of a detected target object. The processor330may extract a feature vector from an image of a target object by using data and program code in the feature vector extraction module344. In an embodiment, the processor330may extract a first feature vector from an image of a target object recognized in the raw image and extract a second feature vector from an image of the target object recognized in the reconstructed image.

The recognition result determination module346is a module for determining whether the target object is correctly recognized in the reconstructed image, based on a degree of overlap between ROIs detected respectively in the raw image and the reconstructed image and a degree of similarity between feature vectors respectively extracted from the raw image and the reconstructed image. The recognition result determination module346may include an overlap degree calculation module346a, a similarity degree calculation module346b, and a comparison module346c.

The overlap degree calculation module346ais a module for calculating the degree of overlap among two or more images and outputting a resulting value. In an embodiment, by using data and program code in the overlap degree calculation module346a, the processor330may calculate a degree of overlap between the first ROI including the target object recognized in the raw image and the second ROI including the target object recognized in the reconstructed image, based on coordinate value information of a bounding box of the first ROI and coordinate value information of a bounding box of the second ROI.

The similarity degree calculation module346bis a module for calculating a degree of similarity between two feature vectors as a numerical value. In an embodiment, by using data and program code in the similarity degree calculation module346b, the processor330may calculate, as a numerical value, a degree of similarity between the first feature vector extracted from the target object detected in the raw image and the second feature vector extracted from the target object detected in the reconstructed image.

The comparison module346cis a module configured to compare the degree of overlap between the first and second ROIs, which is calculated using the overlap degree calculation module346a, with a first threshold, and compare the degree of similarity between the first and second feature vectors, which is calculated using the similarity degree calculation module346b, with a second threshold. The processor330may compare the degree of overlap between the first and second ROIs with the first threshold and compare the degree of similarity between the first and second feature vectors with the second threshold by using data and program code in the comparison module346c.

The compression ratio adjustment module348is a module for determining a compression ratio at which the raw image is compressed to obtain the reconstructed image, based on whether the target object is recognized, which is determined by the recognition result determination module346. In an embodiment, by using data and program code in the compression ratio adjustment module348, the processor330may adjust a value of a quantization parameter, based on a result of comparing the degree of overlap between the first and second ROIs with the first threshold and a result of comparing the degree of similarity between the first and second feature vectors with the second threshold.

Because the object recognition module342, the feature vector extraction module344, the recognition result determination module346, and the compression ratio adjustment module348stored in the memory340respectively correspond to the object recognition module2310, the feature vector extraction module2320, the recognition result determination module2330, and the compression ratio adjustment module2340, descriptions thereof that are already provided above are omitted.

In an embodiment, the processor330may change a value of a compression ratio when the degree of overlap is less than or equal to the first threshold or the degree of similarity is less than or equal to the second threshold. In this case, the processor330may change the value of the compression ratio to a value lower than a compression ratio used to compress and encode the previously reconstructed image. In an embodiment, when the degree of overlap exceeds the first threshold and the degree of similarity exceeds the second threshold, the processor330may determine the compression ratio used to generate the reconstructed image as a final compression ratio.

The processor330may generate a bitstream by compressing the raw image using the determined compression ratio and encoding the compressed raw image. In an embodiment, the processor330may store the generated bitstream in an image data storage350.

The image data storage350is a storage device configured to store image data generated by the processor330. The image data storage350may include a non-volatile memory. For example, the image data storage350may include at least one non-volatile memory among a flash memory-type memory, a hard disk-type memory, a multimedia card micro-type memory, a card-type memory (e.g., an SD card or an XD memory), a magnetic memory, a magnetic disk, and an optical disk.

In an embodiment, the bitstream stored in the image data storage350may be transmitted to the communication interface360according to control by the processor330.

The communication interface360may perform data transmission and reception between the mobile device300and a server200. The communication interface360may transmit or receive data to or from the server200by using a wireless communication method including at least one of a wireless LAN, Wi-Fi, WFD, wireless broadband Internet (WiBro), World Interoperability for Microwave Access (WiMAX), shared wireless access protocol (SWAP), and WiGig. In an embodiment, the processor330may transmit the bitstream to the server200using the communication interface360.

The server200may store the bitstream received from the mobile device300in a storage210.

FIG.2is a diagram illustrating a flow of data through modules included in a device, according to an embodiment of the present disclosure.

In the present disclosure, the ‘device’ may include the IP camera system100or the mobile device300. Hereinafter, for convenience of description, components of the IP camera system100are described as an example of the device.

Referring toFIG.2, a raw image iomay be input to the object recognition module2310and the encoding module1320. The raw image iomay be obtained by capturing an image of a target area or a target object being monitored via the imaging sensor (1100ofFIG.1A).

The raw image iois input to the object recognition module2310, and the object recognition module2310may detect a target object in the raw image io. In an embodiment, the target object may include at least one of a person's face, hairstyle, clothing, and posture, but is not limited thereto. The object recognition module2310may detect a first ROI R1including the detected target object in the raw image io. The first ROI R1may be an image of a target object, e.g., a bounding box including a person's face.

The target object detected in the raw image iois input to the feature vector extraction module2320, and the feature vector extraction module2320may extract a first feature vector f1from the image of the target object. The first feature vector f1may be, for example, a vector including feature values corresponding to landmarks, such as the eyes, the nose, and the mouth of a person's face. However, the first feature vector f1is not limited thereto, and may be a vector including feature values corresponding to at least one key-point extracted from a person's hairstyle, clothing, or a specific posture.

The first ROI R1detected in the raw image ioand the first feature vector f1extracted from the target object may be input to the recognition result determination module2330.

When the raw image iois input to the encoding module1320, the encoding module1320may compress and encode the raw image iousing a preset compression ratio QP0to output encoded image data. In an embodiment, the encoded image data may be quantized transform coefficients.

The encoded image data is input to the decoding module1330, and the decoding module1330may reconstruct residual data by performing inverse quantization and inverse transform on the encoded image data. The decoding module1330may obtain a reconstructed image ireconby adding prediction data obtained via inter-prediction and intra-prediction to the reconstructed residual data.

The reconstructed image ireconis input to the object recognition module2310, and the object recognition module2310may detect a target object in the reconstructed image irecon. The target object detected in the reconstructed image ireconmay be the same as the target object detected in the raw image io, but image quality, such as the resolution and sharpness of an image of the target object, may be lower than that of an image of the target object detected in the raw image io. The object recognition module2310may detect a second ROI R2including the target object detected in the reconstructed image irecon. The second ROI R2may be an image of a target object, e.g., a bounding box including a person's face.

The target object detected in the reconstructed image ireconis input to the feature vector extraction module2320, and the feature vector extraction module2320may extract the second feature vector f2from the image of the target object detected in the reconstructed image irecon. Because the second feature vector f2is the same as the first feature vector f1except that it is extracted from the image of the target object detected in the reconstructed image irecon, descriptions already provided above are omitted.

The second ROI R2detected in the reconstructed image ireconand the second feature vector f2extracted from the target object may be input to the recognition result determination module2330.

The overlap degree calculation module2332may calculate a degree of overlap between the first ROI R1and the second ROI R2, based on coordinate value information of the bounding box of the first ROI R1and coordinate value information of the bounding box of the second ROI R2. In an embodiment, the overlap degree calculation module2332may calculate the degree of overlap between the first ROI R1and the second ROI R2by using an IoU method. The overlap degree calculation module2332may provide information about the calculated degree of overlap to the comparison module2336.

The similarity degree calculation module2334may calculate a degree of similarity between the first feature vector f1and the second feature vector f2. In an embodiment, the similarity degree calculation module2334may calculate the degree of similarity representing a correlation between the first feature vector f1and the second feature vector f2as a numerical value, and provide information about the calculated degree of similarity to the comparison module2336.

The comparison module2336may determine whether the target object is correctly recognized by using the degree of overlap (R1, R2) input from the overlap degree calculation module2332and the degree of similarity (f1, f2) input from the similarity degree calculating module2334. In an embodiment, the comparison module2336may determine whether the target object is correctly recognized in the reconstructed image ireconcomparing a value of the degree of overlap (R1, R2) with a first threshold and comparing a value of the degree of similarity (f1, f2) with a second threshold.

For example, the comparison module2336determines that the target object is not correctly recognized in the reconstructed image ireconwhen the value of the degree of overlap (R1, R2) is less than or equal to the first threshold or the value of the degree of similarity (f1, f2) are less than or equal to the second threshold. In this case, the comparison module2336provides a result of determining whether the target object is recognized to the compression ratio adjustment module2340.

When a result of determination that the target object is not correctly recognized is input from the comparison module2336, the compression ratio adjustment module2340may adjust a compression ratio to a first compression ratio QP1that is lower than the value of the preset compression ratio QP0used to obtain the reconstructed image irecon. The compression ratio adjustment module2340may provide the encoding module1320with information about the value of the first compression ratio QP1that is an adjusted compression ratio.

The encoding module1320may obtain encoded image data by compressing the raw image iousing the value of the first compression ratio QP1input from the compression ratio adjustment module2340and encoding the compressed raw image io. Operations performed by the decoding module1330, the object recognition module2310, the feature vector extraction module2320, the recognition result determination module2330, and the compression ratio adjustment module2340are repeatedly performed as the image data obtained by compressing and encoding the raw image iois input.

In another example, when the value of the degree of overlap (R1, R2) exceeds the first threshold and the value of the degree of similarity (f1, f2) exceeds the second threshold, the comparison module2336may determine that the target object is correctly recognized in the reconstructed image irecon. In this case, the comparison module2336may transmit a signal requesting generation of a bitstream to the encoding module1320.

When the signal requesting generation of the bitstream is input, the encoding module1320may generate a bitstream by compressing the raw image ioat the preset compression ratio QP0and encoding the compressed raw image io.

The encoding module1320may transmit the bitstream to the server200. In this case, the encoding module1320may transmit the bitstream to the server200located outside via the network interface (1400ofFIG.1A).

In a general surveillance camera system, due to constraints on storage capacity or network bandwidths, a video captured using a camera is compressed and encoded using a compression method complying with the standard (e.g., H.264/AVC, HEVC (H.265), etc.) and transmitted to the server200. In this case, when encoding is performed at an excessively high compression ratio, the quality of an image degrades so that a face, hairstyle, clothing, etc. of a person to be recognized may not be identified in the image. Furthermore, in exceptional situations, such as limited network bandwidth, data packet loss due to network instability, an error in an encoding operation, data packet loss in a UDP environment, etc., a decoded image frame degrades, and a person's face and the like cannot be recognized in the degraded image frame. When the person's face cannot be recognized in the degraded image frame, the person is mistakenly recognized as another person, and thus, it is difficult to track an unauthorized person.

The devices according to the embodiments described with reference toFIGS.1A,1B, and2are capable of solving the above-described problems, i.e., increasing a recognition rate of a particular target object (e.g., a face, a hairstyle, clothing, a posture, etc.) even in exceptional situations, such as limited network bandwidth, data packet loss due to network instability, an error in an encoding operation, data packet loss in a UDP environment, etc. According to an embodiment of the present disclosure, the IP camera system100may recognize target objects respectively in a raw image and a reconstructed image obtained from the IP camera1000, extract feature vectors from the target objects, and adjust a compression ratio of the reconstructed image, based on a degree of overlap between ROIs including the target objects and a degree of similarity between the feature vectors, thereby determining a compression ratio optimized for target object recognition. Accordingly, the IP camera system100of the present disclosure may significantly improve a recognition rate of a target object, e.g., a person's face, and reduce a false recognition rate.

FIG.3is a flowchart of a method, performed by a device, of processing an image, according to an embodiment of the present disclosure.

In the present disclosure, the ‘device’ may include the IP camera system100or the mobile device300. Hereinafter, for convenience of description, a case in which the IP camera system100, as an example of the device, performs operations of the method is described. The operations or functions described below may be performed equally by the mobile device300.

In operation S310, the IP camera system100obtains a raw image from at least one imaging sensor (1100ofFIG.1A). The raw image is image data obtained by capturing an image of a target area or target object being monitored via the imaging sensor1100, and is uncompressed image data.

In operation S320, the IP camera system100obtains a reconstructed image by compressing the raw image at a preset compression ratio, encoding the compressed raw image into image data, and decoding the encoded image data. In an embodiment, the IP camera (1000ofFIG.1A) may include the encoder (1310ofFIG.1A), and use the encoder1310to compress raw image at a preset compression ratio, encode the compressed raw image, and decode the encoded image, thereby generating a reconstructed image. In an embodiment, the encoder1310included in the IP camera1000may compress, encode, and decode the raw image by using at least one of the image compression standards, for example, H.264/AVC and HEVC (H.265), to thereby generate a reconstructed image. The IP camera1000may transmit the raw image and the reconstructed image to the processing device (2000ofFIG.1A). The processing device2000may receive image data of the raw image and image data of the reconstructed image from the IP camera1000via the network interface (2100ofFIG.1A).

In operation S330, the IP camera system100detects, in the raw image, a target object and a first ROI including the target object, and extracts a first feature vector from the target object. For example, the target object may include at least one of a person's face, hairstyle, clothing, and posture. In an embodiment, by using a known image processing technique or an AI model including deep learning, the IP camera system100may recognize the target object in the raw image and extract the first feature vector from the recognized target object.

In an embodiment, the processor (2200ofFIG.1A) of the processing device2000may detect a target object in the raw image by performing training via a CNN model. The processor2200may detect the target object in the raw image by using trained model parameters of the CNN model. For example, when the target object is a face, the processor2200may detect a person's face in the raw image by using a large scale dataset, such as CASIA-WebFace, VGGFaceNGGFace 2, or MS-Celeb-1M.

In an embodiment, the processor2200may extract the first feature vector from an image of the target object by using a known image processing technique or an AI model including deep learning. For example, the processor2200may extract the first feature vector from the image of the target object by using an ICA or PCA method.

In an embodiment, the processor2200may extract the first feature vector from the image of the target object by using a CNN model. For example, the processor2200may extract feature values by striding over the image of the target object using a kernel having a predetermined size and predetermined number of channels, obtain a plurality of layers including the extracted feature values, and obtain a feature vector map by applying weights to the plurality of layers. An ReLU model may be used in the process of obtaining a feature vector value, and in order to improve the efficiency of the CNN model, an operation of normalizing feature values through dropout and an operation of performing pooling or max pooling may be further added. The processor2200may extract the first feature vector from the image of the target object detected in the raw image by using a CNN model, such as MobileNet v1/v2.

In operation S340, the IP camera system100recognizes, in the reconstructed image, a target object and a second ROI including the target object, and extracts a second feature vector from the target object. Because the target object is recognized and the second feature vector is extracted from the ‘reconstructed image’ instead of the raw image in operation S340, operation S340is performed in the same manner as operation S330except for only a difference in the image from which the target object is recognized and the second feature vector is extracted. Therefore, descriptions already provided above with respect to operation S330are omitted.

In operation S350, the IP camera system100determines whether the target object is correctly recognized based on a degree of overlap between the first and second ROIs and a degree of similarity between the first and second feature vectors. In an embodiment, the processor2200of the processing device2000may calculate the degree of overlap between the first ROI and the second ROI and the degree of similarity between the first feature vector and the second feature vector. In an embodiment, the processor2200may compare the calculated degree of overlap with a first threshold, compare the calculated degree of similarity with a second threshold, and determine whether the target object is correctly recognized in the reconstructed image based on comparison results.

Here, a ‘compression ratio’ represents an image compression parameter for encoding a raw image by using a method that matches a standard specification. Compression ratios may exist in various forms for each standard specification. For example, in standards for playing video (e.g., H.264/AVC, HEVC (H.265), etc.), a compression ratio of an image may be determined based on a quantization parameter.

A ‘quantization parameter’ is a parameter indicating a degree of quantization of transform coefficients defined in a standard related to compression of video, e.g., H.264/AVC or HEVC (H.265), and may be used to determine a compression ratio of an image. In an embodiment, a compression ratio is proportional to a value of a quantization parameter. For example, when the value of the quantization parameter is large, the compression ratio is high, and when the value of the quantization parameter is small, the compression ratio is low.

In an embodiment, when the degree of overlap is less than or equal to the first threshold or the degree of similarity is less than or equal to the second threshold, the processor2200may determine that the target object is not correctly recognized in the reconstructed image.

In an embodiment, when the degree of overlap exceeds the first threshold and the degree of similarity exceeds the second threshold, the processor2200may determine that the target object is correctly recognized in the reconstructed image.

In operation S360, the IP camera system100may adjust a compression ratio based on a result of determining whether the target object is recognized. In an embodiment, when it is determined that the target object is not correctly recognized in the reconstructed image, the IP camera system100may change a value of the compression ratio to a value lower than a compression ratio used to compress and encode the previously reconstructed image. In an embodiment, when it is determined that the target object is correctly recognized in the reconstructed image, the IP camera system100may determine the compression ratio used to generate the reconstructed image obtained in operation S320as a final compression ratio.

Detailed embodiments of operations S350and S360are described in detail with reference toFIGS.4and5.

FIG.4is a flowchart of a method, performed by a device, of determining whether a target object is recognized in a reconstructed image, according to an embodiment of the present disclosure.

In the present disclosure, the ‘device’ may include the IP camera system100or the mobile device300. Hereinafter, for convenience of description, a case in which the IP camera system100, as an example of the device, performs operations of the method is described. The operations or functions described below may be performed equally by the mobile device300.

In operation S410, the IP camera system100calculates a degree of overlap between the first ROI and the second ROI. In an embodiment, the IP camera system100may obtain position coordinate value information of a bounding box of the first ROI including the target object detected in the raw image and position coordinate value information of a bounding box of the second ROI including the target object detected in the reconstructed image, and calculate the degree of overlap indicating the degree to which the first and second ROIs overlap by using the obtained position coordinate values. In an embodiment, the processor (2200ofFIG.1A) of the processing device (2000ofFIG.1A) may calculate the degree of overlap between the first and second ROIs by using an IoU method for calculating the degree of overlap using position coordinate value information.

In operation S420, the IP camera system100calculates a degree of similarity between the first feature vector and the second feature vector. By using a known method for calculating a correlation between vectors, the processor2200of the processing device2000may calculate the degree of similarity between the first feature vector extracted from the image of the target object detected in the raw image and the second feature vector extracted from the image of the target object detected in the reconstructed image.

InFIG.4, operations S410and S420may be performed simultaneously and independently. However, the present disclosure is not limited thereto, and operation S410may be performed before operation S420, or operation S420may be performed before operation S410.

In operation S430, the IP camera system100may compare the degree of overlap calculated in operation S410with a first threshold α, and compare the degree of similarity calculated in operation S420with a second threshold β. For example, the first threshold α may have a value of 0.8 and the second threshold β may have a value of 0.5, but the first and second thresholds α and β are not limited thereto.

When the degree of overlap exceeds the first threshold α and the degree of similarity exceeds the second threshold β (operation S440), the IP camera system100determines that the target object is correctly recognized in the reconstructed image. When the target object is correctly recognized in the reconstructed image, the method is connected to {circle around (a)} ofFIG.5, and after operation S440ofFIG.4is performed, operation S510ofFIG.5may be performed.

When the degree of overlap is less than or equal to the first threshold α or the degree of similarity is less than or equal to the second threshold β (operation S450), the IP camera system100determines that the target object is not correctly recognized in the reconstructed image. When the target object is correctly recognized in the reconstructed image, the method is connected to {circle around (a)} ofFIG.5, and after operation S440ofFIG.4is performed, operation S510ofFIG.5may be performed.

FIG.5is a flowchart of a method, performed by a device, of determining a compression ratio of an image based on a result of determining whether a target object is recognized, according to an embodiment of the present disclosure.

In the present disclosure, the ‘device’ may include the IP camera system100or the mobile device300. Hereinafter, for convenience of description, a case in which the IP camera system100, as an example of the device, performs operations of the method is described. The operations or functions described below may be performed equally by the mobile device300.

Operations S510to S550shown inFIG.5are concrete operations of operation S360shown inFIG.3. Operations S320, S330, and S340shown inFIG.5are the same as those shown inFIG.3.

In operation S510, the IP camera system100may determine the compression ratio, e.g., a value of a quantization parameter, used to generate the reconstructed image obtained in operation S320, as a final compression ratio. Operation S510may be performed after operation S440ofFIG.4connected thereto through {circle around (a)} is performed.

In operation S520, the IP camera system100changes the compression ratio to a lower value. Operation S520may be performed after operation S450of FIG.4connected thereto through {circle around (b)} is performed. In an embodiment, the IP camera system100may adjust the compression ratio to a value lower than the compression ratio used to generate the reconstructed image obtained in operation S320. For example, the processor (2200ofFIG.1A) of the processing device (2000ofFIG.1A) may change the value of the quantization parameter used to generate the reconstructed image to a lower value.

In operation S530, the IP camera system100provides information about the changed compression ratio to the encoder (1310ofFIG.1A) of the IP camera (1000ofFIG.1A). In an embodiment, the processing device2000may transmit information about the changed value of the quantization parameter to the IP camera1000via the network interface (2100ofFIG.1A).

After operation S530, the IP camera system100repeats operations S320, S330, S340, S350, and S360. In an embodiment, the IP camera1000may generate a reconstructed image based on the information about the compression ratio received from the processing device2000by compressing the raw image at the changed compression ratio, encoding the compressed raw image, and decoding the encoded image. The IP camera system100may repeat at least once operations of receiving the reconstructed image (operation S320), recognizing the target object in the raw image and extracting the first feature vector from the target object (operation S330), recognizing the target object in the received reconstructed image and extracting the second feature vector from the target object (operation S340), determining whether the target object is correctly recognized, based on the degree of overlap and the degree of similarity operation (operation S350), and adjusting a compression ratio (operation S360). The IP camera system100may repeat operations S320, S330, S340, S350, and S360until the degree of overlap exceeds the first threshold a and the degree of similarity exceeds the second threshold β.

FIG.6is a flowchart of an operation method of an IP camera system100and a server200, according to an embodiment of the present disclosure.

Referring toFIG.6, the IP camera system100may include an IP camera1000and a processing device2000.

The IP camera1000may include an imaging sensor1100, an encoding module1320, and a decoding module1330. Because the imaging sensor1100, the encoding module1320, and the decoding module1330are the same as the corresponding components shown inFIG.1A, descriptions already provided above are omitted.

The processing device2000may include an object recognition module2310, a feature vector extraction module2320, a recognition result determination module2330, and a compression ratio adjustment module2340. Because the object recognition module2310, the feature vector extraction module2320, the recognition result determination module2330, and the compression ratio adjustment module2340are the same as the corresponding components shown inFIG.1A, descriptions already provided above are omitted.

In operation S610, the IP camera1000obtains a raw image using the imaging sensor1100. In an embodiment, the IP camera1000may capture an image of a target area or target object being monitored via a lens and obtain a raw image of the target area or target object via the imaging sensor1100. In an embodiment, the imaging sensor1100may be configured as one or a plurality of imaging sensors.

In operation S612, the IP camera1000transmits data of the raw image to the processing device2000. The IP camera1000may transmit image data of the raw image to the processing device2000by using at least one of the wired and wireless communication methods including Ethernet, wired and wireless LANs, Wi-Fi, WFD, and WiGig.

In operation S620, the IP camera1000generates a reconstructed image by compressing the raw image using information about a compression ratio, encoding the compressed raw image, and decoding the encoded image. The IP camera1000may compress and encode the raw image using the encoding module1320. The encoding module1320may be configured to encode the raw image at a preset compression ratio and obtain encoded image data from the raw image. In an embodiment, by using the encoding module1320, the IP camera1000may obtain prediction data from the raw image via inter-prediction and intra-prediction, obtain residual data produced by subtracting the prediction data from the raw image, and obtain encoded image data by performing transformation and quantization on the residual data. For example, the encoded image data may be quantized transform coefficients.

By using the decoding module1330, the IP camera1000may generate a reconstructed image by decoding the encoded image data. In an embodiment, by using the decoding module1330, the IP camera1000may reconstruct the residual data by performing inverse quantization and inverse transform on the encoded image data, i.e., the quantized transform coefficients, and obtain a reconstructed image by adding the prediction data obtained via inter-prediction and intra-prediction to the reconstructed residual data.

In operation S622, the IP camera1000transmits data of the reconstructed image to the processing device2000. Like in operation S612, the IP camera1000may transmit image data of the reconstructed image to the processing device2000by using at least one of the wired and wireless communication methods including Ethernet, wired and wireless LANs, Wi-Fi, WFD, and WiGig.

In operation S630, the processing device2000detects a target object and a first ROI including the target object in the raw image by using the object recognition module2310, and extracts a first feature vector from the target object by using the feature vector extraction module2320. For example, the target object may include at least one of a person's face, hairstyle, clothing, and posture. In an embodiment, by using a known image processing technique or an AI model including deep learning, the processing device2000may recognize the target object in the raw image and extract the first feature vector from the recognized target object.

Because operation S630is the same as operation S330shown inFIG.3, descriptions already provided above are omitted.

In operation S640, the processing device2000recognizes a target object and a second ROI including the target object in the reconstructed image by using the object recognition module2310, and extracts a second feature vector from the target object by using the feature vector extraction module2320. Because the target object is recognized and the second feature vector is extracted from the ‘reconstructed image’ instead of the raw image in operation S640, operation S640is performed in the same manner as the operation S630except for only a difference in the image from which the target object is recognized and the second feature vector is extracted. Therefore, descriptions already provided above with respect to operation S630are omitted.

In operation S650, the processing device2000determines whether the target object is correctly recognized, based on a degree of overlap between the first and second ROIs and a degree of similarity between the first and second feature vectors. In an embodiment, the processor (2200ofFIG.1A) of the processing device2000may calculate the degree of overlap between the first ROI recognized in the raw image and the second ROI recognized in the reconstructed image by using the recognition result determination module2330. In an embodiment, the processor2200may calculate the degree of similarity between the first feature vector extracted from the target object in the raw image and the second feature vector extracted from the target object in the reconstructed image by using the recognition result determination module2330. In an embodiment, the processor2200may compare the calculated degree of overlap with a first threshold, compare the calculated degree of similarity with a second threshold, and determine whether the target object is correctly recognized in the reconstructed image based on comparison results.

When it is determined that the target object is correctly recognized in the reconstructed image (operation S60), the processing device2000may determine the compression ratio used to generate the reconstructed image as a final compression ratio. In an embodiment, by using the compression ratio adjustment module2340, the processor2200of the processing device2000may obtain information about the compression ratio used when generating a reconstructed image by compressing and encoding the raw image in operation S620and determine a value of the obtained compression ratio as a final compression ratio.

In operation S662, the processing device2000transmits information about the final compression ratio to the IP camera1000. In an embodiment, the processor2200may transmit information about the final compression ratio to the IP camera1000via the network interface (2100ofFIG.1A).

In operation S670, the IP camera1000transmits a bitstream to the server200. In an embodiment, the IP camera1000may generate a bitstream by compressing the raw image at the final compression ratio determined based on the information about the final compression ratio received from the processing device2000, and encoding the compressed raw image. The IP camera1000may transmit the generated bitstream to the server200. In this case, the IP camera1000may transmit the bitstream to the server200located outside via the network interface (1400ofFIG.1A).

When it is determined that the target object is not correctly recognized in the reconstructed image (operation S680), the processing device2000changes the compression ratio to a low value. In an embodiment, by using the compression ratio adjusting module2340, the processor2200may change the value of the compression ratio to a value lower than the compression ratio used to generate the reconstructed image by compressing the raw image and encoding the compressed raw image in operation S620.

In operation S682, the processing device2000transmits information about the changed compression ratio to the IP camera1000. In an embodiment, the processing device2000may transmit information about the compression ratio changed to a low value to the IP camera1000via the network interface2100.

When the IP camera1000receives the information about the changed compression ratio in operation S682, the IP camera1000may perform operations S620and S622again, and the processing device2000may perform operations S630, S640, and650again. When the target object is not correctly recognized in the reconstructed image in operation S650(If No), the processing device2000may perform operations S680and S682again. The processing device2000may repeat n times the operations of changing the value of the compression ratio to a lower value and transmitting information about the changed compression ratio to the IP camera until it is determined that the target object is correctly recognized in the reconstructed image in operation S650.

FIG.7is a diagram illustrating a method, performed by a device, of recognizing a target object according to a coding unit (CU) and adjusting a compression ratio for each CU based on a recognition result, according to an embodiment of the present disclosure.

In the present disclosure, the ‘device’ may include the IP camera system100or the mobile device300. Hereinafter, for convenience of description, a case in which the IP camera system100, as an example of the device, performs operations of the method is described. The operations or functions described below may be performed equally by the mobile device300.

Referring toFIG.7, the processing device (2000ofFIG.1A) of the IP camera system100may split each of the raw image700and the reconstructed image710obtained by the IP camera (1000ofFIG.1A) according to a CU having a preset image size. In an embodiment, each CU may include a preset number of pixels. For example, each CU may include 16×16 or 64×64 pixels, and is not limited thereto. The processing device2000may split the raw image700into a plurality of CUs700-1to700-n. Similarly, the processing device2000may split the reconstructed image710into a plurality of CUs710-1to710-n. In an embodiment, the processing device2000may split each of the raw image700and the reconstructed image710in units of an LCU. For example, an LCU is a unit of coding including at least one CU.

The processing device2000may identify at least one first CU700-a,700-b, and700-cin which a target object is recognized from among the plurality of CUs700-1to700-nincluded in the raw image700. In an embodiment, by using instructions or program code related to the object recognition module2310, the processor (2200ofFIG.1A) of the processing device2000may detect target objects in the raw image700and identify the at least one first CU700-a,700-b, and700-cin which the target objects are respectively detected from among the plurality of CUs700-1to700-n. A target object may include, for example, at least one of a person's face, hairstyle, clothing, and posture, but is not limited thereto. AlthoughFIG.7shows the number of at least one first CUs700-a,700-b, and700-cis 3, the number of CUs in which the target object is detected is not limited to 3.

The processor2200may identify bounding boxes of the target objects respectively detected in the at least one first CU700-a,700-b, and700-c, and obtain position coordinate values of the identified bounding boxes. In an embodiment, the processor2200may respectively set the bounding boxes identified in the at least one first CU700-a,700-b, and700-cto be at least one first ROI R1-a, R1-b, and R1-c.

By using the instructions or program code related to the object recognition module2310, the processor2200may detect target objects in the reconstructed image710and identify the at least one second CU710-a,710-b, and710-cin which the target objects are respectively detected from among the plurality of CUs710-1to710-n. By using the object recognition module2310, the processor2200may identify bounding boxes of the target objects respectively detected in the at least one second CUs710-a,710-b, and710-c, and respectively set the identified bounding boxes to be at least one second ROI R2-a, R2-b, and R2-c.

The processing device2000may respectively extract first feature vectors from images of the target objects included in the at least one first ROI R1-a, R1-b, and R1-c. In an embodiment, the processor2200may respectively extract the first feature vectors from the images of the target objects included in the at least one of the first ROI R1-a, R1-b, and by using instructions or program code related to the feature vector extraction module2320. The feature vector extraction module2320may be configured to extract a feature vector from an image of a target object by using a known image processing method or an AI model including deep learning. For example, the feature vector extraction module2320may be configured to extract a feature vector from an image of a target object by using an ICA or PCA method.

In an embodiment, the processor2200may respectively extract second feature vectors from images of the target objects included in the at least one second ROI R2-a, R2-b, and R2-cby using the feature vector extraction module2320.

The processing device2000may determine a result of recognizing a target object, based on a degree of overlap between each of the at least one first ROI R1-a, R1-b, and a corresponding one of the at least one second ROI R2-a, R2-b, and R2-c, and a degree of similarity between the first and second feature vectors corresponding thereto In an embodiment, the processor2200may calculate the degree of overlap between the at least one first ROI R1-a, R1-b, R1-cand the at least one second ROI R2-a, R2-b, R2-cby using instructions or program code related to the recognition result determination module2330. In an embodiment, the processor2200may calculate the degree of overlap between the at least one first ROI R1-a, R1-b, and R1-cand the at least one second ROI R2-a, R2-b, and R2-cby using an IoU method.

The processor2200may calculate the degree of similarity between the first and second feature vectors by using the recognition result determination module2330. In an embodiment, by using the recognition result determination module2330, the processor2200may calculate, as a numerical value, a correlation between the first feature vector extracted from the image of the target object included in the 1-a-th ROI R1-aand the second feature vector extracted from the image of the target object included in the 2-a-th ROI R2-a. Similarly, the processor2200may calculate, as a numerical value, a correlation between the first feature vector extracted from the image of the target object included in the 1-b-th ROI R1-band the second feature vector extracted from the image of the target object included in the 2-b-th ROI R2-b.

By using the recognition result determination module2330, the processor2200may determine whether a target object is accurately recognized by comparing the degree of overlap between the at least one first ROI R1-a, R1-b, and and the at least one second ROI R2-a, R2-b, and R2-cwith a first threshold and comparing the degree of similarity between the first and second feature vectors with a second threshold. The processor2200may determine, based on comparison results, the CU710-bin which the target object is not correctly recognized from among the at least one second CU710-a,710-b, and710-c. In an embodiment, the processor2200may determine, as a CU in which the target object is not recognized, the CU710-bfor which the calculated degree of overlap is less than or equal to the first threshold or the degree of similarity is less than or equal to the second threshold from among the at least one second CU710-a,710-b, and710-c.

The processing device2000adjusts a value of a compression ratio of the determined CU710-bin which the target object is not recognized. In an embodiment, the processor2200may change the value of the compression ratio of the CU710-bin which the target object is not recognized to a lower value by using instructions or program code related to the compression ratio adjustment module2340.

In an embodiment, the processor2200may determine a value of a compression ratio used to generate the reconstructed image710as a final compression ratio for the 2-a-th CU710-aand the 2-c-th CU710-cthat are determined as CUs in which the target objects are correctly recognized because the degree of overlap exceeds the first threshold and the degree of similarity exceeds the second threshold as a result of comparison via the recognition result determination module2330.

FIG.8is a flowchart of a method, performed by a device, of recognizing a target object according to a CU and adjusting a compression ratio for each CU based on a recognition result, according to an embodiment of the present disclosure.

In the present disclosure, the ‘device’ may include the IP camera system100or the mobile device300. Hereinafter, for convenience of description, a case in which the IP camera system100, as an example of the device, performs operations of the method is described. The operations or functions described below may be performed equally by the mobile device300.

In operation S810, the IP camera system100splits each of the raw image and the reconstructed image according to a CU. In an embodiment, the processing device (2000ofFIG.1A) of the IP camera system100may split each of the raw image and the reconstructed image obtained by the IP camera (1000ofFIG.1A) according to a CU having a preset image size. In an embodiment, the processing device2000may split each of the raw image and the reconstructed image in units of an LCU.

In operation S820, the IP camera system100may identify, in each of the raw image and the reconstructed image, at least one CU in which a target object is recognized. In an embodiment, the processing device2000may identify at least one first CU in which a target object is recognized from among a plurality of CUs included in the raw image. Similarly, the processing device2000may identify at least one second CU in which target objects are respectively recognized from among a plurality of CUs included in the reconstructed image.

In operation S830, the IP camera system100determines a result of recognition of a target object for each of the identified at least one CU, based on information about a degree of overlap between ROIs in the raw image and the reconstructed image and a degree of similarity between feature vectors. In an embodiment, the processing device2000may identify bounding boxes for images of the target objects respectively included in the at least one first CU identified in the raw image and set the identified bounding boxes to be at least one first ROI. The processing device2000may identify bounding boxes for images of the target objects respectively included in the at least one second CU identified in the reconstructed image and set the identified bounding boxes to be at least one second ROI. In an embodiment, the processing device2000may calculate the degree of overlap between the at least one first ROI and the at least one second ROI. In an embodiment, the processing device2000may calculate the degree of overlap between the at least one first ROI and the at least one second ROI by using an IoU method.

The processing device2000may extract a first feature vector from the at least one first ROI and a second feature vector from the at least one second ROI. The processing device2000may calculate a degree of similarity between the first feature vector and the second feature vector. In an embodiment, the processing device2000may calculate a correlation between the first and second feature vectors as a numerical value.

The processing device2000may determine whether a target object is accurately recognized by comparing the degree of overlap between the at least one first ROI and the at least one second ROI with a first threshold and comparing the degree of similarity between the first feature vector and the second feature vector with a second threshold.

In operation S840, the IP camera system100determines a CU in which the target object is not recognized, based on the recognition result determined in operation S830. In an embodiment, the processing device2000may identify a CU for which the degree of overlap is less than or equal to the first threshold and the degree of similarity is less than or equal to the second threshold as a result of the comparison in operation S830, from among the at least one second CU identified in the reconstructed image, and determine the identified CU as a CU in which the target object is not correctly recognized.

In operation S850, the IP camera system100adjusts a compression ratio of the CU in which the target object is not recognized. In an embodiment, the processing device2000may change the compression ratio of the CU in which the target object is not recognized to a value lower than a value of a compression ratio of the reconstructed image in operation S810.

FIG.9is a diagram illustrating a method, performed by a device, of recognizing a target object910in an image frame900and extracting a feature vector from the recognized target object910, according to an embodiment of the present disclosure.

In the present disclosure, the ‘device’ may include the IP camera system100or the mobile device300. Hereinafter, for convenience of description, a case in which the IP camera system100, as an example of the device, performs operations of the method is described. The operations or functions described below may be performed equally by the mobile device300.

Referring toFIG.9, the processing device (2000ofFIG.1A) of the IP camera system100may recognize the target object910in the image frame900by using an object recognition module2310. The image frame900may be one of a plurality of frames constituting a video. A target object may be, for example, a person's face, hairstyle, clothing, or posture, but is not limited thereto. In the embodiment shown inFIG.9, the target object910may be a person's posture.

For example, the object recognition module2310may detect the target object910in the image frame900by performing training using trained model parameters of a CNN model. However, a method, performed by the object recognition module2310, of detecting the target object910is not limited to the above-described method. In an embodiment, the object recognition module2310may recognize the target object910in the image frame900by using an AI model including at least one of an RNN model, SVM, linear regression, logistic regression, Naive Bayes classification, random forest, decision tree, and a k-nearest neighbor algorithm.

In an embodiment, the processing device2000may identify a bounding box including the target object910detected in the image frame900by using the object recognition module2310. The processing device2000may set the identified bounding box to be an ROI R910.

The processing device2000may extract a feature vector from an image of the target object910included in the ROI R910by using the feature vector extraction module2320. In an embodiment, the processing device2000may identify at least one key-point KP1to KPnfrom the image of the target object in the ROI R910, extract coordinate values of each of the identified at least one key-point KP1to KPn, and convert the extracted coordinate values into feature values according to a preset rule. The processing device2000may obtain a feature vector by using the feature values.

In the embodiment shown inFIG.9, by using the feature vector extraction module2320, the processing device2000may identify the first to n-th key-points KP1to KPnas representative values that may represent the person's posture, respectively convert the identified first to n-th key-points KP1to KPninto feature values, and generate a feature vector using the feature values.

In another embodiment, the feature vector extraction module2320may store a preset value according to each of a person's hairstyle (e.g., hair color, hair length, etc.), clothing color, and clothing style (e.g., T-shirt, jeans, dress, skirt, etc.). By using the feature vector extraction module2320, the processing device2000may obtain a preset value related to a hairstyle or clothing from an image of a target object, and extract a feature vector using the obtained preset value.

A program executed by the IP camera system100described through the present disclosure may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. A program may be executed by any system capable of executing computer-readable instructions.

Software may include a computer program, a piece of code, an instruction, or a combination of one or more thereof, and may configure a processing device to operate as desired or instruct the processing device independently or collectively.

The software may be implemented as a computer program including instructions stored in computer-readable storage media. Examples of the computer-readable recording media include magnetic storage media (e.g., ROM, RAM, floppy disks, hard disks, etc.), optical recording media (e.g., CD-ROM, digital versatile disks (DVDs), etc.), etc. The computer-readable recording media may be distributed over computer systems connected through a network so that computer-readable code may be stored and executed in a distributed manner. The media may be read by a computer, stored in a memory, and executed by a processor.

A computer-readable storage medium may be provided in the form of a non-transitory storage medium. In this regard, the term ‘non-transitory’ only means that the storage medium does not include a signal and is a tangible device, and the term does not differentiate between where data is semi-permanently stored in the storage medium and where the data is temporarily stored in the storage medium.

Furthermore, programs according to embodiments disclosed in the present specification may be included in the form of a computer program product when provided. The computer program product may be traded, as a product, between a seller and a buyer.

The computer program product may include a software program and a computer-readable storage medium having stored thereon the software program. For example, the computer program product may include a product (e.g., a downloadable application) in the form of a software program electronically distributed by a manufacturer of a device or through an electronic market (e.g., Google Play Store™ and App Store™). For such electronic distribution, at least a part of the software program may be stored on the storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a server of the manufacturer, a server of the electronic market, or a relay server for temporarily storing the software program.

In a system consisting of a server and the device, the computer program product may include a storage medium of the server or a storage medium of the device. Alternatively, in a case where there is a third device (e.g., a smartphone) communicatively connected to the server or the device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the device or the third device or that is transmitted from the third device to the device.

In this case, one of the server, the device, and the third device may execute the computer program product to perform methods according to embodiments of the disclosure. Alternatively, two or more of the server, the device, and the third device may execute the computer program product to perform the methods according to the embodiments of the disclosure in a distributed manner.

For example, the server may execute the computer program product stored therein to control the device communicatively connected to the server to perform the methods according to the embodiments of the disclosure.

In another example, the third device may execute the computer program product to control the device communicatively connected to the third device to perform the methods according to the embodiments of the disclosure.

In a case where the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute the computer program product that is pre-loaded therein to perform the methods according to the embodiments of the disclosure.

While the embodiments have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various modifications and changes in form and details may be made from the above descriptions. For example, adequate effects may be achieved even when the above techniques are performed in a different order than that described above, and/or the aforementioned components such as computer systems or modules are coupled or combined in different forms and modes than those described above or are replaced or supplemented by other components or their equivalents.