ELECTRONIC DEVICE PERFORMING SCALING USING ARTIFICIAL INTELLIGENCE MODEL AND METHOD FOR OPERATING THE SAME

An electronic device is provided. The electronic device includes memory, a camera module, a communication module, and at least one processor operatively connected to the memory, the camera module, and the communication module. The memory, when executed by the at least one processor, cause the electronic device to establish a call connection with a network based on the communication module, identify a first image captured based on the camera module, identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device, identify a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model, and transmit the second image through the call connection.

TECHNICAL FIELD

The disclosure relates to an electronic device that performs scaling using an artificial intelligence model (AI) model and a method for operating the same.

BACKGROUND ART

When transmitting multimedia content, the multimedia content (e.g., an image) may be encoded by a codec that complies with data compression standards. A bitstream generated as a result of the encoding may be transmitted through a communication channel. For example, when an electronic device establishes a connection for a video call, a bitstream may be transmitted through the connection for a call.

To downsize the bitstream, multimedia content, e.g., an image, may be down-scaled. The down-scaled image may have a relatively smaller data size than the original image. The down-scaled image may be encoded, and a bitstream generated as a result of the encoding may have a relatively smaller data size than the bitstream corresponding to the original image. The receiving electronic device may receive the bitstream and then decode it using a codec. The receiving electronic device may up-scale the decoding result. By the up-scaling, a higher-resolution image than the image generated as a result of decoding may be generated and/or provided. An AI model for down-scaling and/or up-scaling may be used for the down-scaling and/or up-scaling.

DISCLOSURE OF INVENTION

Solution to Problems

Aspects of the disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the disclosure is to provide an electronic device that performs scaling using an AI model and a method for operating the same.

In accordance with an aspect of the disclosure, an electronic device is provided. The electronic device includes memory, a camera module, a communication module, and at least one processor operatively connected to the memory, the camera module, and the communication module. The memory, when executed by the at least one processor, cause the electronic device to establish a call connection with a network based on the communication module, identify a first image captured based on the camera module, identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device, identify a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model, and transmit the second image through the call connection based on the communication module.

In accordance with another aspect of the disclosure, a method for operating an electronic device is provided. The method includes establishing a call connection with a network based on a communication module, identifying a first image captured based on a camera module of the electronic device, identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device, identifying a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model, and transmitting the second image through the call connection based on a communication module of the electronic device.

According to an embodiment of the disclosure, one or more non-transitory computer-readable storage media storing at least one computer-readable instruction is provided. The at least one instruction, when executed by at least one processor of an electronic device, configures the electronic device to perform at least one operation including establishing a call connection with a network based on a communication module, identifying a first image captured based on a camera module of the electronic device, identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device, identifying a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model, and transmitting the second image through the call connection based on the communication module.

In accordance with another aspect of the disclosure, an electronic device is provided. The electronic device includes memory, a display module, a communication module, and at least one processor operatively connected to the memory, the display module, and the communication module The memory, when executed by the at least one processor, cause the electronic device to establish a call connection with a network based on the communication module, receive the first image through the call connection based on the communication module, identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device, identify a second image corresponding to the first image output from an artificial intelligence model for up-scaling, trained to receive information associated with a low-resolution image and a bitrate as an input value to output a high-resolution image, by inputting the first image and the first information to the artificial intelligence model, and control the display module to display at least a portion of the second image.

In accordance with another aspect of the disclosure, a method for operating an electronic device is provided. The method includes establishing a call connection with a network based on a communication module, receiving a first image through the call connection based on the communication module of the electronic device, identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device, identifying a second image corresponding to the first image output from an artificial intelligence model for up-scaling, trained to receive information associated with a low-resolution image and a bitrate as an input value to output a high-resolution image, by inputting the first image and the first information to the artificial intelligence model, and controlling a display module of the electronic device to display at least a portion of the second image.

According to an embodiment of the disclosure, one or more non-transitory computer-readable storage media storing at least one computer-readable instruction is provided. The at least one instruction, when executed by at least one processor of an electronic device, configures the electronic device to perform at least one operation including receiving a first image through a call connection with a network based on a communication module of the electronic device, identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device, identifying a second image corresponding to the first image output from an artificial intelligence model for up-scaling, trained to receive information associated with a low-resolution image and a bitrate as an input value to output a high-resolution image, by inputting the first image and the first information to the artificial intelligence model, and controlling a display module of the electronic device to display at least a portion of the second image.

In accordance with another aspect of the disclosure, an electronic device for training a first AI model for down-scaling and a second AI model for up-scaling is provided. The electronic device includes memory and at least one processor. The memory, when executed by the at least one processor, cause the electronic device to identify training data including a first image, which is a high-resolution image, and first information associated with a bitrate, identify a second image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model, identify a third image, which is a high-resolution image, output from the second AI model, based on inputting the second image and the first information to the second AI model, identify a fourth image by down-scaling the first image, identify a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and train at least a portion of the first AI model and the second AI model based on the total loss.

In accordance with another aspect of the disclosure, a method for training a first AI model for down-scaling and a second AI model for up-scaling is provided. The method includes identifying training data including a first image which is a high-resolution image and first information associated with a bitrate, identifying a second image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model, identifying a third image, which is a high-resolution image, output from the second AI model, based on inputting the second image and the first information to the second AI model, identifying a fourth image by down-scaling the first image, identifying a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and training at least a portion of the first AI model and the second AI model based on the total loss.

According to an embodiment of the disclosure, one or more non-transitory computer-readable storage media storing at least one computer-readable instruction is provided. The at least one instruction, when executed by at least one processor of an electronic device, configures the electronic device to perform at least one operation including identifying training data including a first image, which is a high-resolution image, and first information associated with a bitrate, identifying a second image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model for down-scaling, identifying a third image, which is a high-resolution image, output from the second AI model, based on inputting the second image and the first information to the second AI model for up-scaling, identifying a fourth image by down-scaling the first image identifying a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and training at least a portion of the first AI model and the second AI model based on the total loss.

MODE FOR THE INVENTION

Referring toFIG.1, an electronic device101in a network environment100may communicate with an external electronic device102via a first network198(e.g., a short-range wireless communication network), or an external electronic device104or a server108via a second network199(e.g., a long-range wireless communication network). According to an embodiment of the disclosure, the electronic device101may communicate with the external electronic device104via the server108. According to an embodiment of the disclosure, the electronic device101may include a processor120, memory130, an input module150, a sound output module155, a display module160, an audio module170, a sensor module176, an interface177, a connecting terminal According to an embodiment of the disclosure, the display module160may include a first display module351corresponding to the user's left eye and/or a second display module353corresponding to the user's right eye., a haptic module179, a camera module180, a power management module188, a battery189, a communication module190, a subscriber identification module (SIM)196, or an antenna module197. In an embodiment of the disclosure, at least one (e.g., the connecting terminal178) of the components may be omitted from the electronic device101, or one or more other components may be added in the electronic device101. According to an embodiment of the disclosure, some (e.g., the sensor module176, the camera module180, or the antenna module197) of the components may be integrated into a single component (e.g., the display module160).

The camera module180may capture a still image or moving images. According to an embodiment of the disclosure, the camera module180may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module188may manage power supplied to the electronic device101. According to an embodiment of the disclosure, the power management module188may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

FIG.2is a view illustrating operations of an electronic device according to an embodiment of the disclosure.

Referring toFIG.2, according to an embodiment of the disclosure, the electronic device101(e.g., the processor120) may establish a connection250(or may be referred to as a session or a channel) for a video call, based on the communication module190. For example, the electronic device101may perform at least one procedure for establishing the connection250for a call according to the Internet protocol (IMS) multimedia subsystem (IMS) standard through a network200but is not limited thereto. The electronic device101may be a mobile origination (MO) device or a mobile termination (MT) device, and is not limited thereto.

According to an embodiment of the disclosure, the electronic device101may identify an image212captured based on the camera module180. For example, the electronic device101may control the display module160to display the captured image (or down-scaled image)212, but is not limited thereto. As is described below, the electronic device101may down-scale the captured image212and may generate a bitstream by encoding the down-scaled image. The electronic device101may transmit the generated bitstream to the external electronic device220through the connection250. The external electronic device220may receive the bitstream. As is described below, the external electronic device220may decode the received bitstream and up-scale the decoded image. The external electronic device220may control a display module221to display an up-scaled image223. Meanwhile, the external electronic device220may also control the display module221to display an image224captured by the camera module (not shown). The external electronic device220may generate a bitstream by encoding the down-scaled image of the captured image224. The external electronic device220may transmit the generated bitstream to the electronic device101through the connection250. The electronic device101may decode the received bitstream. The electronic device101may perform up-scaling on the decoded image and may control the display module160to display an image211based on the up-scaling result. Accordingly, the display module160may display the image212captured by the camera module180of the electronic device101and the image211transmitted from the external electronic device220. The external electronic device220may display the captured image224and the image223transmitted from the electronic device101.

According to an embodiment of the disclosure, the electronic device101may down-scale the captured image using an AI model for down-scaling. For example, the AI model may be trained to receive a high-resolution image and information associated with a bitrate as input values and output a low-resolution image (or referred to as a down-scaled image). According to an embodiment of the disclosure, the electronic device101may up-scale the received and decoded image using the AI model for up-scaling. For example, the AI model may be trained to receive a low-resolution image and information associated with a bitrate as input values and output a high-resolution image (or an up-scaled image). The structure and/or training of the AI model for down-scaling and/or the AI model for up-scaling is described below.

FIG.3Ais a view illustrating a comparative example according to an embodiment of the disclosure.

Referring toFIG.3A, at least some of the operations according to the comparative example based onFIG.3Aand/or another comparative example of the disclosure may be performed by the electronic device101according to an embodiment of the disclosure.

According to a comparative example, the electronic device101may identify a high-resolution image301captured by the camera module180. The high-resolution image301may have, e.g., a video graphic array (VGA)-class resolution or a high definition (HD)-class resolution, but this is exemplary and the resolution of the high-resolution image301is not limited thereto. A filter310operated by the electronic device101may down-scale the high-resolution image301to output a low-resolution image302. The filter310, as, e.g., a normal filter, may perform down-scaling based on a Bicubic method or a lanczos method, but the down-scaling method is not limited thereto. The low-resolution image302may have, e.g., a quarter VGA (QVGA)-class resolution or an nHD-class resolution, but this is illustrative and the resolution of the low-resolution image302is not limited thereto.

An encoder311operated by the electronic device101may generate a bitstream by encoding the low-resolution image302. The encoder311may perform encoding using a codec (e.g., moving picture experts group 2 (MPEG-2), H.264, MPEG-4, high efficiency video coding (HEVC), VC-1, VP8, VP9, or AV1), but the type of the codec is not limited. The bitstream may be packetized by, e.g., a real-time transport protocol (RTP), and transmitted. A network prediction module313operated by the electronic device101may predict a communication environment between the electronic device101and the network200. The network prediction module313may predict the communication environment between the electronic device101and the network200based on a network parameter (e.g., one-way delay, perceived bitrate, and/or packet loss rate). The prediction of the communication environment between the electronic device101and the network200by the network prediction module313is described below. The bitrate for encoding may be set based on a communication environment prediction result between the electronic device101and the network200. For example, when it is predicted that the communication environment between the electronic device101and the network200is relatively good, the bitrate may be set to be relatively high, but this is merely an example and is not limited thereto. When the bitrate is determined, the remaining codec parameters (e.g., resolution and/or framerate (or frames per second (FPS)) for encoding may be determined. For example, when the bitrate is determined based on the communication environment between the electronic device101and the network200, the resolution and/or the framerate may be determined based on the determined bitrate and the compression rate of the codec. The bitstream generated as a result of the encoding of the encoder311may be provided to the communication module190aof the receiving electronic device through the communication module190. The received bitstream may be decoded by a decoder320operated by the receiving electronic device (which may be the same as the electronic device101). A decoded image323may be rendered by a renderer321operated by the receiving electronic device, and accordingly, at least a portion of the decoded image323may be displayed on the receiving electronic device. Meanwhile, according to the comparative example ofFIG.3A, the decoded image323may have the same resolution as the down-scaled low-resolution image302. Accordingly, an image having a relatively small resolution may be displayed on the receiving electronic device.

FIG.3Bis a view illustrating a comparative example according to an embodiment of the disclosure.

Referring toFIG.3B, at least some of the operations according to the comparative example may be performed by the electronic device101according to an embodiment of the disclosure.

According to a comparative example, the electronic device101may identify a high-resolution image301captured by the camera module180. A high-resolution image331may have, e.g., a VGA-class resolution or an HD-class resolution, but this is illustrative and the resolution of the high-resolution image331is not limited thereto. A down scaler314operated by the electronic device101may down-scale the high-resolution image331to output a low-resolution image332. The down scaler314may be implemented as, e.g., an AI model, but is not limited as long as down-scaling may be performed. When implemented as an AI model, the down scaler314may be referred to as an AI scaler. The low-resolution image332may have, e.g., a QVGA)-class resolution or an nHD-class resolution, but this is illustrative and the resolution of the low-resolution image332is not limited thereto. The encoder311operated by the electronic device101may generate a bitstream by encoding the low-resolution image332. A network prediction module313operated by the electronic device101may predict a communication environment between the electronic device101and the network200. The bitrate for encoding may be set based on a communication environment prediction result between the electronic device101and the network200. For example, when the bitrate is determined based on the communication environment between the electronic device101and the network200, the resolution and/or the framerate may be determined based on the determined bitrate and the compression rate of the codec. The bitstream generated as a result of the encoding of the encoder311may be provided to the communication module190aof the receiving electronic device through the communication module190. The received bitstream may be decoded by the decoder320operated by the receiving electronic device (which may be the same as the electronic device101).

An up scaler335may up-scale the decoded image332to provide a high-resolution image334. The up scaler335may be implemented as, e.g., an AI model, but is not limited as long as up-scaling may be performed. When implemented as an AI model, the up scaler335may be referred to as an AI scaler. The high-resolution image334may have substantially the same resolution as the high-resolution image331captured by, e.g., the transmitting electronic device101. The high-resolution image334may be rendered by a renderer321operated by the receiving electronic device, and accordingly, at least a portion of the high-resolution image334may be displayed on the receiving electronic device. Meanwhile, in another example, as illustrated inFIG.3A, the receiving electronic device101may render the decoded low-resolution image332without the up-scaling process.

As described above, a high-resolution image having substantially the same resolution as the image captured by the transmitting electronic device101may be provided by the receiving electronic device. Further, since the codec parameters (e.g., bitrate, resolution, and/or framerate) of the encoder311may be set based on the communication environment between the electronic device101and the network200, if the communication environment between the electronic device101and the network200is poor, a low-quality bitstream may be transmitted, thereby preventing delay or loss. However, in the example ofFIG.3B, the communication environment between the electronic device101and the network200is not considered during down-scaling and/or up-scaling. Because the quality (e.g., whether it is blocky) on the receiving side is affected by the encoded bitrate transmitted, it may be required to introduce an AI scaler that considers the bitrate in real time (or semi-real time). In embodiments of the disclosure, e.g., an AI model in which information associated with a bitrate set based on the communication environment between the electronic device101and the network200is considered may be used for down-scaling and/or up-scaling, and/or may be trained.

FIG.4is a flowchart illustrating operations of an electronic device according to an embodiment of the disclosure. The embodiment ofFIG.4is described with reference toFIG.5.

FIG.5is a view illustrating a transmitting electronic device and a receiving electronic device according to an embodiment of the disclosure.

Referring toFIGS.4and5together, according to an embodiment of the disclosure, the electronic device101(e.g., the processor120) may establish a call connection with a network, based on the communication module190, in operation401. As described above, e.g., the electronic device101may perform a procedure according to the IMS standard, but this is exemplary, and the procedure for establishing a call connection is not limited. In operation403, the electronic device101may identify a first image501captured based on the camera module180. The first image501is, e.g., a high-resolution image may have a VGA-class resolution or an HD-class resolution, but this is illustrative and the resolution of the high-resolution image is not limited thereto. In operation405, the electronic device101may identify the first information associated with the first bitrate corresponding to the first image501, based on the communication environment between the network and the electronic device101. In one example, bit per pixel (BPP), which is information associated with the bitrate, may be expressed as Equation 1.

The bitrate in Equation 1 may be determined based on, e.g., the communication environment. For example, a relatively high bitrate may be determined when the communication environment is relatively good, and a relatively low bitrate may be determined when the communication environment is relatively poor, but the disclosure is not limited thereto. For example, the communication environment may be categorized into a plurality of ranges, and bitrates may be mapped and managed for each category, but this is exemplary, and there is no limitation on a method for determining an indicator (or format) indicating the communication environment and/or a bitrate corresponding to the indicator. Embodiments related to the communication environment are described below. When the bitrate is determined, resolution and/or framerate, which are the remaining codec parameters, may be determined. For example, the resolution and/or framerate corresponding to the bitrate may be determined based on the codec compression rate, but this is exemplary and the determination method is not limited thereto. In one example, the communication environment may be determined by the network prediction module313. The bit rate corresponding to the communication environment may be determined by at least one of the network prediction module313or the encoder311. The remaining codec parameters (e.g., resolution and/or framerate) corresponding to the bitrate may be determined by at least one of the network prediction module313or the encoder311. The bitrate-related information (e.g., BPP as shown in Equation 1) may be determined by at least one of the network prediction module313or the encoder311. Meanwhile, the operation of the network prediction module313and/or the encoder311may be performed by, e.g., the processor120, but is not limited thereto.

According to an embodiment of the disclosure, in operation407, the electronic device101may identify a second image502corresponding to the first image501output from a first AI model510by inputting the first image501and the first information (e.g., BPP) to the first AI model510for down-scaling. In contrast to the down scaler314in the comparative example ofFIG.3Bconfigured to receive only the high-resolution image331as an input value and provide a corresponding low-resolution image332, the first AI model510ofFIGS.4and5may be configured to receive not only the first image501but also information (e.g., BPP) related to the bitrate as an input value and provide the second image502having a low-resolution. The first AI model510may include, e.g., a neural network for extracting an image feature corresponding to the first image501and a neural network for extracting a meta information feature corresponding to information (e.g., a BPP) associated with the bitrate, and may have a structure for performing a multiplication operation between the image feature and the meta information feature, but is not limited thereto, and a description thereof and training of the first AI model510are described below. In operation409, the electronic device101may transmit the second image502through a call connection, based on the communication module190. Here, the transmission of the second image502may include, e.g., generation of a bitstream based on encoding of the second image502and transmission of the bitstream. As shown inFIG.5, the encoder311may encode the second image502to generate a bitstream. As described above, the encoder311may generate a bitstream by performing encoding based on the bitrate determined based on the communication environment and the resolution and/or framerate set based on the bitrate. As described above, at least some of codec parameters including the bitrate, the resolution, and/or the framerate may be used not only by the encoder311, but also as at least some of the input values of the first AI model510for down-scaling. The bitstream may be provided to the receiving electronic device (which may be the same device as the electronic device101) through the communication module190. If a plurality of AI models are configured for various bit rates, respectively, to reflect a change in the communication environment in real time, and any one of the plurality of AI models is selected to perform down-scaling, the size of information (e.g., a library) to be stored in the electronic device101may increase sharply. In contrast, the electronic device101according to an embodiment of the disclosure may perform down-scaling using an AI model trained to receive information associated with the bitrate and the high-resolution image as input values and output the low-resolution image corresponding to the high-resolution image, so that the amount of information of the AI model may be relatively small as compared to when the plurality of AI models are configured for various bitrates, respectively. Meanwhile, the operation of the receiving electronic device ofFIG.5is described with reference toFIG.6. As described above, even when the call channel is in a relatively poor state, high-quality content may be provided to the receiving device without deterioration of content quality (low-resolution/locky/delay).

FIG.6is a flowchart illustrating operations of an electronic device according to an embodiment of the disclosure. The embodiment ofFIG.6is described with reference toFIG.5.

Referring toFIGS.5and6together, according to an embodiment of the disclosure, the electronic device101(e.g., the processor120) may establish a call connection with a network, based on the communication module190, in operation601. Meanwhile,FIG.5illustrates that the receiving electronic device includes a communication module190a. However, the electronic device101according to an embodiment may perform the operation of the receiving electronic device ofFIG.5. The electronic device101may establish a call connection, e.g., based on performing a procedure according to the IMS standard, but the method of establishment is not limited. In operation603, the electronic device101may receive the second image505through a call connection, based on the communication module190. Here, the reception of the second image505may include, e.g., reception of a bitstream generated as the second image is encoded and decoding of the bitstream by the decoder320. Operations performed by the decoder320, a network prediction module515, and/or the renderer321may be performed by the processor120of the electronic device101, but are not limited thereto. In operation605, the electronic device101may identify second information associated with the second bitrate. For example, the second bitrate estimated by the network prediction module515and/or the decoder320and the second information (e.g., BPP) corresponding to the second bitrate may be identified. The identification of the second information is described below.

According to an embodiment of the disclosure, in operation607, the electronic device101may identify the third image507corresponding to the second image505output from a second AI model512by inputting the second image505and the second information (e.g., BPP) to the second AI model512for up-scaling. In operation609, the electronic device101may display the third image507(or at least a portion thereof). The second AI model512may include, e.g., a neural network for extracting an image feature corresponding to the second image505and a neural network for extracting a meta information feature corresponding to information (e.g., a BPP) associated with the bitrate, and may have a structure for performing a multiplication operation between the image feature and the meta information feature, but is not limited thereto, and a description thereof and training of the second AI model512are described below. If a plurality of AI models are configured for various bit rates, respectively, to reflect a change in the communication environment in real time, and any one of the plurality of AI models is selected to perform up-scaling, the size of information (e.g., a library) to be stored in the electronic device101may increase sharply. In contrast, the electronic device101according to an embodiment of the disclosure may perform up-scaling using an AI model trained to receive information associated with the bitrate and the low-resolution image as input values and output the high-resolution image corresponding to the low-resolution image, so that the amount of information of the AI model may be relatively small as compared to when the plurality of AI models are configured for various bitrates, respectively.

FIG.7illustrates VMAF scores according to a comparative example according to an embodiment of the disclosure.

Referring toFIG.7, it illustrates a video multimethod assessment fusion (VMAF) score701for the bitrate when an AI model for down-scaling to receive the information (e.g., BPP) associated with the bitrate and the high-resolution image as input values and output the low-resolution image and an AI model for up-scaling to receive the information (e.g., BPP) associated with the bitrate and the low-resolution image as input values and output the high-resolution image are used according to an embodiment. VMAF may be, e.g., an objective overall reference video quality metric developed by the University of Southern California, the IPI/LS2N Research Institute of the University of Nantes, the Image and Video Engineering Research Institute of the University of Nantes, and Netflix, which is exemplary and is not limited to the evaluation score indicating image quality. A VMAF score702for the bitrate when an AI model for down-scaling to receive the high-resolution image as an input value and output the low-resolution image and an AI model for up-scaling to receive the low-resolution image as an input value and output the high-resolution image are used is illustrated. A VMAF score703for a case where an AI model according to the comparative example is not used is illustrated. It may be identified that the VMAF score701according to the embodiment is higher than the VMAF scores702and703for other cases.

FIG.8Ais a view illustrating an AI model for down-scaling according to an embodiment of the disclosure.

Referring toFIG.8A, according to a comparative example, the electronic device101may identify a first image801(or an image included in the training data set) captured by the camera module180. The electronic device101may down-scale the first image801to a second image803using an AI model for down-scaling. Meanwhile, inFIG.8A, a process of applying an AI model has been described, but the embodiment ofFIG.8Amay be performed during a training process, which is described with reference toFIGS.8C and8D. In the comparative example ofFIG.8A, the AI model may have, e.g., the structure of ResNet. Accordingly, the mobile electronic device101for a video call may use the AI model with a relatively small amount of computation, but this is merely an example, and it will be understood by one of ordinary skill in the art that the AI model is not limited as long as it is structured for down-scaling. Meanwhile, ResNet may be trained to enhance overall resolution by enhancing the residual line rather than image scaling, but is not limited thereto. For example, the Raw YUV420method may be used, and training may be performed such that while UV based on legacy scaling is generated, the Y (luma) channel is enhanced, but the disclosure is not limited thereto. The AI model according to the comparative example may include, but is not limited to, a portion810for Bicubic down-scaling, a portion812for image feature extraction and enhancement/residual image configuration, and an image adder for Bilinear up-scaling814. The portion810(or AI model) for Bicubic down-scaling may perform down-scaling based on, e.g., the Bicubic method, but the down-scaling method is not limited thereto. Based on the Bicubic method, e.g., the second image803having a resolution ¼ times that of the first image801may be generated, but is not limited thereto. The portion810for down-scaling may be a portion in ResNet except for a portion corresponding to Residual. The portion812for image feature extraction and enhancing/residual image configuration is a portion corresponding to the residual in ResNet, and may include, e.g., a convolution layer, but is not limited thereto, and may include a plurality of sub-AI structures. The image adder for Bilinear up-scaling814may be an adder capable of adding the original image (the down-scaled original image in this comparative example) defined in ResNet and the image corresponding to residual. Meanwhile, in the comparative example, a CLIP function as in Equation 2 may be used for the output image (e.g., the second image803).

Output in Equation 2 may be the image (e.g., the second image803inFIG.8A) output from ResNet. By the CLIP function, “downscale+Residual”, e.g., the summation of the result of the portion810and the result of the portion812may be adjusted between the minimum pixels (MIN pixel) and maximum pixels (MAX pixel). As described above, according to the comparative example, a second image803that is a down-scaled image based on ResNet may be provided. However, as described above, the AI model according to the comparative example does not use the information related to the bitrate.

FIG.8Bis a view illustrating an AI model for up-scaling according to an embodiment of the disclosure.

Meanwhile, inFIG.8B, a process of applying an AI model has been described, but the embodiment ofFIG.8Bmay be performed during a training process, which is described with reference toFIGS.8C and8D.

Referring toFIG.8B, according to the comparative example, the electronic device101may up-scale the second image803to a third image805using an AI model for up-scaling. The AI model according to the comparative example may include, but is not limited to, a portion for Bilinear up-scaling814, a portion816for image feature extraction and enhancement/residual image configuration, and an image adder818. The portion (or AI model) for Bilinear up-scaling814may perform up-scaling based on, e.g., the Bilinear method, but the up-scaling method is not limited thereto. Based on the Bilinear method, e.g., the third image805having a resolution 4 times that of the second image803may be generated, but is not limited thereto. The portion for Bilinear up-scaling may be a portion in ResNet except for a portion corresponding to Residual. The portion816for image feature extraction and enhancing/residual image configuration is a portion corresponding to the residual in ResNet, and may include, e.g., a convolution layer, but is not limited thereto, and may include a plurality of sub-AI structures. The adder818may be an adder capable of adding the original image (the up-scaled original image in this comparative example) defined in ResNet and the image corresponding to residual. As described above, according to the comparative example, a third image805that is an up-scaled image based on ResNet may be provided. However, as described above, the AI model according to the comparative example does not use the information related to the bitrate. For example, the AI model for down-scaling described with reference toFIG.8Aand the AI model for up-scaling described with reference toFIG.8Bmay be trained together, which is described with reference toFIGS.8C and8D.

FIG.8Cis a flowchart illustrating training an AI model for down-scaling and an AI model for up-scaling according to an embodiment of the disclosure. The embodiment ofFIG.8Cis described with reference toFIG.8D.

FIG.8Dis a view illustrating training an AI model for down-scaling and an AI model for up-scaling according to an embodiment of the disclosure.

According to a comparative example and/or an embodiment of the disclosure, training of at least one AI model may be performed by a trainer. The training may be performed, e.g., by the server108(or may be another computing device) and/or by the electronic device101executing the AI model. It may be understood that the operation performed by the trainer in the disclosure is performed by the electronic device101and/or the server108.

Referring toFIGS.8C and8Dtogether, in operation831, the trainer may identify the second image803by inputting the first image801to a first AI model821for down-scaling. The first AI model821may be, e.g., the ResNet described with reference toFIG.8A, but is not limited thereto. In operation832, the trainer may identify the third image805by inputting the second image803to a second AI model823for up-scaling. The second AI model823may be, e.g., the ResNet described with reference toFIG.8B, but is not limited thereto. In operation833, the trainer may identify a first loss Loss1 (Up-Similarity) based on the similarity between the first image801and the third image805. In operation834, the trainer may identify a fourth image807obtained by up-scaling the second image803. For example, the trainer may identify the fourth image807based on an up scaler825using the lanczos method, but is not limited thereto. In operation835, the trainer may identify the second loss Loss2 (Legacy-Similarity) based on the similarity between the first image801and the fourth image807. In operation836, the trainer may train the first AI model821and the second AI model823based on the first loss Loss1 and the second loss Loss2. For example, the total loss may be as shown in Equation 3.

In Equation 3, α and β may be weights. The trainer may perform training to minimize total losses. As described above, the first AI model821for down-scaling and the second AI model823for up-scaling may be trained together. The loss and/or calculation of the loss may be based on, e.g., a mean square error (L2) loss, a negative structural similarity index (SSIM) loss, or an absolute error after Gaussian filter (GL1) loss, but this is exemplary and the type is not limited thereto.

FIG.8Eis a flowchart illustrating training an AI model for down-scaling and an AI model for up-scaling according to an embodiment of the disclosure. The embodiment ofFIG.8Eis described with reference toFIG.8F.

FIG.8Fis a view illustrating training an AI model for up-scaling according to an embodiment of the disclosure.

Referring toFIGS.8E and8Ftogether, in operation851, the trainer may identify a second image873by inputting a first image871to a first AI model872for fixed down-scaling. In the comparative example ofFIGS.8E and8F, the parameter of the first AI model872may be set to have a fixed value. For example, the parameter of the first AI model872may be determined based on the training described with reference toFIGS.8C and8D, and since the parameter of the first AI model872is not additionally trained in the embodiments ofFIGS.8E and8F, it will be understood by one of ordinary skill in the art that the word “fixed” is used. In operation852, the trainer may encode the second image873using an encoder874. In operation853, the trainer may identify a third image876by decoding the encoded second image using a decoder875. For example, encoding and/or decoding may use a fixed QP value. In actual video call streaming, a constant/variable bitrate mode (CBR/VBR) may be used, and the bitrate may be changed in real time according to the communication environment. In operation854, the trainer may identify a fourth image878by inputting the third image876to a second AI model877for up-scaling. In operation855, the trainer may identify a loss Loss1 (Up-Similarity) based on the similarity between the first image871and the fourth image878. In operation856, the trainer may train the second AI model877based on the loss Los1. The trainer may train the second AI model877to minimize the loss. For example, the trainer may perform training set based onFIGS.8C and8Dand/or training set based onFIGS.8E and8F, thereby training the AI model for down-scaling and/or the AI model for up-scaling.

FIG.9Ais a view illustrating an AI model for down-scaling according to an embodiment of the disclosure.

Referring toFIG.9A, according to an embodiment of the disclosure, the electronic device101may identify a first image901captured (or selected from the training data set) by the camera module180. The electronic device101may down-scale the first image901to a second image902using an AI model for down-scaling. Meanwhile, inFIG.9A, a process of applying an AI model has been described, but the embodiment ofFIG.9Amay be performed during a training process, which is described with reference toFIGS.9C and9D. In the embodiment ofFIG.9A, the AI model may have, e.g., the structure of ResNet. Meanwhile, ResNet may be trained to enhance overall resolution by enhancing the residual line rather than image scaling, but is not limited thereto.

The AI model according to an embodiment of the disclosure may include, but is not limited to, a portion911for Bicubic down-scaling, a portion912for image feature extraction, an image multiplier913, a portion914for enhancing/residual image configuration, a portion915for extracting information features associated with the bitrate, and an image adder916. The portion911(or AI model) for Bicubic down-scaling may perform down-scaling based on, e.g., the Bicubic method, but the down-scaling method is not limited thereto. Based on the Bicubic method, e.g., the second image902having a resolution ¼ times that of the first image901may be generated, but is not limited thereto. The portion911for down-scaling may be a portion in ResNet except for a portion corresponding to Residual. The image feature extractor912may include, e.g., at least one convolution layer for extracting a feature, but this is merely an example, and it will be understood by one of ordinary skill in the art that implementation of the image feature extractor912is not limited thereto, and other neural networks, such as an RNN may also be used.

According to an embodiment of the disclosure, the portion915for extracting the information feature associated with the bitrate may be configured to receive the information (e.g., BPP) associated with the bitrate as an input value and output the feature. Meanwhile, other values other than the BPP may be implemented as input values to the portion915, and input information to the portion915may be referred to as meta information. The meta information may include, e.g., the BPP as information associated with the bitrate, but this is merely an example and may also include, but is not limited to, the specifications of the camera module180, the location where the video call is performed, the mode of the camera module180(e.g., the front photographing mode or the rear photographing mode), the network state, the network type, whether lighting is used during call, and/or video frame-related information (e.g., the face-to-face video frame, the roadside video frame, the multi-person video frame, and no-person video frame, but not limited thereto). For example, the portion915for extracting the information feature associated with the bitrate may include at least one fully-connected layer. The portion915for extracting the information feature associated with the bitrate may be implemented as a dense network, but this is exemplary and the type thereof is not limited. The multiplier913may cross-multiply the output of the portion912and the output of the portion915. The portion914for enhancing/residual image configuration may receive the cross-multiplication result, perform enhancement/residual configuration, and output the result. The adder916may add the output from the portion914for enhancing/residual image configuration to the output from the portion911for down-scaling, and thus the second image902may be output. In contrast to the AI model described with reference toFIG.8A, the AI model described with reference toFIG.9Amay receive the high-resolution image (e.g., the first image901) and the value (e.g., BPP) associated with the bitrate as input values, and may output the low-resolution image (e.g., the second image902). Accordingly, training on various bitrates (or codec parameters) may be performed, and an AI model for an environment in which codec parameters are changed according to a change in the communication environment may be provided. The AI model may be trained to enhance the residual line, but is not limited thereto. For example, to enhance the residual line, the residual line may be used for the luma (Y) channel, but is not limited thereto. For example, upon encoding in a relatively low bitrate range, a bitstream of a relatively lower quality than a relatively high bitrate range is generated, and thus a residual line of up-scaling in a relatively low bitrate range may be supposed to have a stronger effect.

FIG.9Bis a view illustrating an AI model for up-scaling according to an embodiment of the disclosure.

Referring toFIG.9B, according to an embodiment of the disclosure, the electronic device101may identify the second image902having a relatively low resolution. The electronic device101may up-scale the second image902to the third image903using an AI model for up-scaling. Meanwhile, inFIG.9B, a process of applying an AI model has been described, but the embodiment ofFIG.9Bmay be performed during a training process, which is described with reference toFIGS.9C and9D. In the embodiment ofFIG.9B, the AI model may have, e.g., the structure of ResNet. Meanwhile, ResNet may be trained to enhance overall resolution by enhancing the residual line rather than image scaling, but is not limited thereto.

According to an embodiment of the disclosure, the AI model for up-scaling may include a portion921for Bilinear up-scaling, a portion922for image feature extraction, a multiplier923, a portion924for enhancing/residual image configuration, a portion925for extracting information features associated with the bitrate, and an adder926. The portion921for Bilinear up-scaling may up-scale the second image902to output the up-scaled image. The up-scaled image may have resolution four times higher than that of the second image902, but is not limited thereto, and the Bilinear method is also exemplary. The portion922for image feature extraction and/or the portion924for enhancing/residual image configuration may include at least one convolution layer, but this is not limited thereto. The portion925for extracting the information feature associated with the bitrate may receive, e.g., information (e.g., BPP) associated with the bitrate and output a feature corresponding thereto. The portion925may include, e.g., a fully-connected layer, but is not limited thereto. The portion925may be implemented as, e.g., a dense network, but is not limited thereto. Meanwhile, other values other than the BPP may be implemented as input values to the portion925, and input information to the portion925may be referred to as meta information. The multiplier923may cross-multiply the output of the portion922and the output of the portion925. The portion924for enhancing/residual image configuration may receive the cross-multiplication result, perform enhancement/residual configuration, and output the result. The adder926may add the output from the portion924for enhancing/residual image configuration to the output from the portion921for up-scaling, and thus the third image903may be output. In contrast to the AI model described with reference toFIG.8B, the AI model described with reference toFIG.9Bmay receive the low-resolution image (e.g., the second image902) and the value (e.g., BPP) associated with the bitrate as input values, and may output the high-resolution image (e.g., the third image903). Accordingly, training on various bitrates (or codec parameters) may be performed, and an AI model for an environment in which codec parameters are changed according to a change in the communication environment may be provided. Meanwhile, as described with reference toFIGS.8C and8D, the AI model for up-scaling and the AI model for down-scaling may be trained together, and this is described with reference toFIGS.9C and9D.

FIG.9Cis a flowchart illustrating training an AI model for down-scaling and an AI model for up-scaling according to an embodiment of the disclosure. The embodiment ofFIG.9Cis described with reference toFIG.9D.

FIG.9Dis a view illustrating training an AI model for down-scaling and an AI model for up-scaling according to an embodiment of the disclosure.

Referring toFIGS.9C and9Dtogether, in operation931, the trainer may identify a second image943by inputting a first image941and the first information to the first AI model942for down-scaling. The first AI model942may be, e.g., the ResNet described with reference toFIG.9A, but is not limited thereto. As described with reference toFIG.9A, the first AI model942may receive first information together with the high-resolution image (e.g., the first image941) as input values. The first information may be, e.g., meta information including BPP, which is information associated with the bitrate, and is not limited thereto. In operation932, the trainer may identify a third image945by inputting the second image943and the first information to a second AI model944for up-scaling. The second AI model944may be, e.g., the ResNet described with reference toFIG.9B, but is not limited thereto. As described with reference toFIG.9B, the second AI model944may receive first information together with the low-resolution image (e.g., the second image943) as input values.

In operation933, the trainer may identify a first loss Loss1 (Up-Similarity) based on the similarity between the first image941and the third image945. In operation934, the trainer may identify the fourth image947obtained by down-scaling the first image941. For example, the trainer may identify the fourth image947based on a down scaler946to downscale the first image941using the lanczos method, but is not limited thereto. In operation935, the trainer may identify a fifth image949obtained by enhancing the fourth image based on the first information associated with the bitrate. For example, the trainer may output the fifth image949using an enhancer948based on the first information, but is not limited thereto, and an enhancing process is described below. In operation936, the trainer may identify the second loss Loss2 (Legacy-Similarity) based on the similarity between the second image943and the fifth image949. In operation937, the trainer may train the first AI model942and the second AI model944based on the first loss Loss1 and the second loss Loss2. For example, the total loss may be as shown in Equation 3 described above. The trainer may perform training to minimize total losses. As described above, the first AI model942for down-scaling and the second AI model944for up-scaling may be trained together. The loss and/or calculation of the loss may be based on, e.g., a mean square error (L2) loss, a negative structural similarity index (SSIM) loss, or an absolute error after Gaussian filter (GL1) loss, but this is exemplary and the type is not limited thereto. For training, e.g., supervised learning in a mini batch gradient decent method may be used, but is not limited thereto. Each of the training data used for each training session may include various resolutions, framerates, and/or bitrates, and accordingly, AI models robust to codec parameters that change according to the network environment may be provided. For example, a perceptual filter may be used in the enhancing process of the enhancer948. As the perceptual filter is used, an effect of changing the performance of the codec according to the state of the input image may be expressed. As the perceptual filter is used, the quality of encoding may be enhanced. If an image is provided, any one of an adaptive weighted average (AWA), a threshold bilateral (TBil), or a just noticeable-distortion (JND) profiled motion-compensated residue, which is a pre-encoding optimizer filter, may be used as the perceptual filter, but is not limited thereto. The goal of the training may be, e.g., that the result of performing down-scaling and up-scaling is substantially the same (or similar) to that for the original image. The goal of the training may be substantially the same (or similar) to, e.g., a result of down-scaling by a down scaler of an AI model of the related art for down-scaling.

FIG.9Eis a flowchart illustrating training an AI model for down-scaling and an AI model for up-scaling according to an embodiment of the disclosure. The embodiment ofFIG.9Eis described with reference toFIG.9F.

FIG.9Fis a view illustrating training an AI model for up-scaling according to an embodiment of the disclosure.

Referring toFIGS.9E and9Ftogether, in operation951, the trainer may identify a second image964by inputting a first image961and first information963to the first AI model963for fixed down-scaling. InFIGS.9E and9F, a second AI model968for up-scaling may be trained, and the parameter of the first AI model963may be set to have the fixed value. In operation952, the trainer may encode the second image964using an encoder965, based on the first information associated with the bitrate. In operation953, the trainer may identify a third image967by decoding the encoded second image using a decoder966. For example, encoding and/or decoding may use a fixed quantization parameter (QP) value. In operation954, the trainer may identify a fourth image969by inputting the third image967and a first information962associated with the bitrate to the second AI model968for up-scaling. In operation955, the trainer may identify the fifth image972obtained by enhancing (971) the first image961based on the first information962associated with the bitrate. In operation956, the trainer may identify a first loss Loss1 (Up-similarity) based on the similarity between the first image961and the fourth image969and a second loss Loss2 (Enhanced-image-Similarity) based on the similarity between the fourth image969and the fifth image972. In operation957, the trainer may train the second AI model968based on the first loss Loss1 and the second loss Loss2. For example, the total loss may be expressed as Equation 3 described above, and β may be 1-α. α may be set based on the BPP, but is not limited. The trainer may train the second AI model968to minimize the total loss. For example, the trainer may perform training set based onFIGS.9C and9Dand/or training set based onFIGS.9E and9F, thereby training the AI model for down-scaling and/or the AI model for up-scaling. The tool for enhancing971is described with reference toFIG.10. Training may be performed with the aim of becoming similar to the original video frame in a relatively high bitrate range, and because codec loss is already high in a relatively low bitrate range, training for enhancing codec loss may be performed. Meanwhile, for the training ofFIGS.9C and9D, the loss function may use the distance-based metric L1L2, L1, or L2, and for the training ofFIGS.9E and9F, the loss function may use the similarity measurement method of SSIM or GL1, but this is not limited thereto.

FIG.10is a view illustrating image enhancing according to an embodiment of the disclosure.

Referring toFIG.10, according to an embodiment of the disclosure, the trainer may smooth an image1001using a Gaussian filter1002. The Gaussian filter1002may smooth the image1001, and thus a smoothed image1003may be provided. An enhancing tool1004may provide an enhanced image1005using the smoothed image1003and the image1001. For example, the enhanced image1005may be represented by Equation 4.

In Equation 4, k may be a value between [0.0, 10.0], and k may be set such that the score (e.g., VMAF) has a maximum value. Meanwhile, the above-described enhancing method is merely exemplary, and it will be understood by one of ordinary skill in the art that the enhancing method is not limited. As described above, the enhanced image1005may be provided, and as described with reference toFIG.9F, the enhanced image (the fifth image972) may be used in training. As the enhanced image is used for training, an AI model capable of providing an output closer to the original image may be provided.

FIG.11Ais a flowchart illustrating a method of operating an electronic device according to an embodiment of the disclosure. The embodiment ofFIG.11Ais described with reference toFIG.11B.

FIG.11Bis a view illustrating a communication environment according to an embodiment of the disclosure.

Referring toFIGS.11A and11B, according to an embodiment of the disclosure, the electronic device101(e.g., the processor120) may identify at least one parameter in operation1101. In operation1103, the electronic device101may predict a communication environment based on at least one parameter. In operation1105, the electronic device101may identify the bitrate, the resolution, and/or the framerate based on the prediction result. For example, the electronic device101may predict the communication environment based on the RTCP, set a relatively high bitrate when the communication environment is relatively good, and set a relatively low bitrate when the communication environment is relatively bad. The bitrate may be set in real time (or semi-real time), and the corresponding framerate and/or resolution may also be set in real time (or semi-real time).

For example, the electronic device101may be required to predict a bandwidth allowed by the network, determine a bitrate within the allowable value, and transmit a packet. For example, the electronic device101may predict a bandwidth based on a parameter to be fed back based on the RTCP. In one example, the communication environment may be classified into three states1123,1124, and1125as shown inFIG.11B. The first state1123may be referred to as, e.g., an “unloaded state”, and a delay, a packet loss, and/or a packet drop may not occur in the first state1123. The second state1124may be referred to as, e.g., a “loaded state”, and in the second state1124, the load may be close to the bandwidth allowed by the network or exceed by a threshold value or less. In the second state1124, e.g., fluctuation of the delay may occur, or a relatively large delay may occur. For example, when a relatively large delay is identified, a repeated increase/decrease in delay is identified, and/or a packet loss of a relatively low level is identified, the communication environment may be identified as the second state1124, but is not limited thereto. The third state1125may be referred to as, e.g., a “congested state”, and in the third state1125, a relatively large number of packet drops may occur. For example, a bottleneck may occur in the entity of the network, and accordingly, when the load exceeds the allowed bandwidth, a continuous delay may occur, or a relatively large packet loss may occur. The packet loss rate may be identified based on the loss fraction in the RR of the RTCP. For example, the instantaneous RTT1121and the smoothed RTT1122may have relatively small values in the first state1123, and may also have relatively small variations. The instantaneous RTT1121and the smoothed RTT1122may have a relatively larger value in the second state1124than in the first state1123. The instantaneous RTT1121and the smoothed RTT1122may continuously increase, e.g., in the third state1125, if there is no timeout. In a state1126in which the congestion is resolved, the instantaneous RTT1121and the smoothed RTT1122may be reduced. Table 1 shows an example of classifying the state of a communication environment for each parameter. Meanwhile, the example of Table 1 may be set to differ for each network (e.g., for each 4G, 5G, and WIFI), but is not limited thereto.

As shown in Table 1, when the one-way delay measured at the current time point is less than 1.2 times the previous one-way delay (prevOWD), the state may be classified as the first state1123, and when the one-way delay is 1.2 times or more, the state may be classified as the second state1124or the third state1125. One-way delay may be predicted based on, e.g., RTT. When the communication environment is relatively poor, the one-way delay may be increased. The RTT may be calculated based on information about the RTCP SR and/or RR. Meanwhile, whether it is less than 1.2 times the prevOWD is merely exemplary, and the numerical value is not limited, or whether it is in the first state1123may be determined depending on whether it is less than the absolute value (e.g., 50 ms) of the delay.

Meanwhile, when the communication environment is relatively good, the total amount of sending bits and the total amount of receiving bits may be the same. However, when the communication environment is relatively poor, the total amount of receiving bits may be lower than the total amount of sending bits. As shown in Table 1, when the perceived bitrate is the same as the sending bitrate, it may be classified as the first state1132. When the perceived bitrate is smaller than the sending bitrate, it may be classified as the second state1124or the third state1125. The perceived bitrate may refer to an actual bitrate reaching the other side, and when the bandwidth is limited, the perceived bitrate may be highly likely to have a limited bandwidth value. As shown in Table 3, when there is no packet loss, the state may be classified as the first state1132. When the packet loss rate is less than or equal to the threshold ratio (e.g., 5%) or the packet loss rate is within a designated threshold period (e.g., three cycles), the state may be classified as the second state1124. For example, when the packet loss rate exceeds the threshold ratio (e.g., 5%) or the packet loss is out of the designated threshold period (e.g., three cycles), the state may be classified as the third state1125. The packet loss rate may be calculated based on the lost fraction information about the RTCP RR.

FIG.12Ais a view illustrating image transmission by an electronic device according to an embodiment of the disclosure.

Referring toFIG.12A, according to an embodiment of the disclosure, the electronic device101(e.g., the processor120) may execute an AI scaling manager1203. The AI scaling manager1203may provide information (e.g., BPP) (or may be referred to as meta information) associated with the bitrate, which is a part of input values to an AI model1205for down-scaling. A network prediction module1201may provide, e.g., a communication environment or a bitrate corresponding to the communication environment, as described with reference toFIGS.11A and11B. The AI scaling manager1203may identify the bitrate corresponding to the communication environment provided from the network prediction module1201or may identify the bitrate provided from the network prediction module1201. The AI scaling manager1203may identify information (e.g., BPP) associated with the bitrate, based on the identified bitrate. For example, the AI scaling manager1203may be provided with camera parameters including the framerate and/or resolution of the camera module180. The AI scaling manager1203may determine the BPP as information associated with the bitrate, based on, e.g., the bitrate, the framerate, or the resolution, but this is merely an example, and it will be understood by one of ordinary skill in the art that information affected by the bitrate may be used as information (or meta information) associated with the bitrate. The AI model1205for down-scaling may receive the image (e.g., the high-resolution image) provided from the camera module180and information (e.g., the BPP) related to the bitrate provided from the AI scaling manager1203as input values and output the low-resolution image. An encoder1207may encode the low-resolution image provided from the AI model1205for down-scaling to provide the encoded image (or bitstream), which may be transmitted through the communication module190. The encoder1207may perform encoding using, e.g., codec parameters including the bitstream. For example, the bitrate may be set to a previously used bitrate.

According to an embodiment of the disclosure, in one example, the AI scaling manager1203may identify the BPP input to the AI model1205, based on the bitrate provided from the network prediction module1201(or identified based on the provided information). In this case, as described in connection with Equation 1, the BPP may be determined as a value obtained by dividing the current bitrate by the product of the framerate and the resolution. Meanwhile, in another example, the AI scaling manager1203may identify a value obtained by dividing the average of the sizes of a designated number (e.g., K which may be a natural number of 1 or more) of encoded images by the resolution as the BPP, which may be expressed as Equation 5.

In Equation 5, “average encoded size” may be the average of the sizes of the designated number of (K) encoded images, and “resolution” may be the resolution.

The AI scaling manager1203may select one of the BPP (e.g., the BPP according to Equation 1) associated with the communication environment or the BPP (e.g., the BPP according to Equation 5) associated with the average of the sizes of the encoded images and provide the selected BPP to the AI model1205. In an example, when the number of the accumulated encoded images is less than a designated number K, the AI scaling manager1203may select the BPP (e.g., the BPP according to Equation 1) associated with the communication environment. In an example, the AI scaling manager1203may select the BPP (e.g., the BPP according to Equation 1) associated with the communication environment when the bitrate identified based on the network prediction module1201changes sharply (or when the communication environment changes sharply or packet loss is large). Meanwhile, the above-described selection conditions of the BPP are exemplary and are not limited thereto.

FIG.12Bis a view illustrating image reception by an electronic device according to an embodiment of the disclosure.

Referring toFIG.12B, according to an embodiment of the disclosure, upon receiving the bitstream, the electronic device101(e.g., the processor120) may decode the received bitstream using a decoder1217. When receiving the bitstream, the electronic device101may execute a AI scaling manager1213. The AI scaling manager1213may provide information (e.g., BPP) (or may be referred to as meta information) associated with the bitrate, which is a part of input values to a AI model1215for up-scaling. The bitrate is information used in the encoding process, and was not previously used by the receiving side. However, the electronic device101according to an embodiment may use the bitrate-related information as a part of the input values to the AI model1215. Accordingly, even when receiving the bitstream, the electronic device101may identify information (e.g., BPP) associated with the bitrate. For example, the electronic device101may identify the bitrate based on information from a network prediction module1211. The AI scaling manager1213may identify the BPP using the bitrate, the identified framerate, and resolution. The AI scaling manager1213may provide the identified BPP to the AI model1215for up-scaling. The AI model1215may receive the BPP provided from the AI scaling manager1213and the low-resolution image provided from the decoder1217as input values, and output the up-scaled high-resolution image. A renderer1219may render the high-resolution image. For example, the bitrate may be set to a previously used bitrate. For example, the bitrate may be set based on the bandwidth at the receiving side measured by the network prediction module1211. For example, the bitrate initially used by the encoder1207may be shared.

According to an embodiment of the disclosure, the AI scaling manager1213of the receiving side may identify information (e.g., BPP) related to the bitstream in any one of the plurality of methods. For example, the AI scaling manager1213may select one of the BPP (e.g., the BPP according to Equation 1) associated with the communication environment or the BPP (e.g., the BPP according to Equation 5) associated with the average of the sizes of the encoded images and provide the selected BPP to the AI model1215. In an example, when the number of the accumulated encoded images is less than a designated number K, the AI scaling manager1213may select the BPP (e.g., the BPP according to Equation 1) associated with the communication environment. The decoder1217may provide the received encoded frame size information to the AI scaling manager1213, and accordingly, the AI scaling manager1213may identify the BPP (e.g., the BPP according to Equation 5) based on the information about the size of the encoded frame. In an example, the AI scaling manager1213may select the BPP (e.g., the BPP according to Equation 1) associated with the communication environment when the bitrate identified based on the network prediction module1201changes sharply (or when the communication environment changes sharply). In an example, when the packet loss exceeds a designated threshold loss value, the AI scaling manager1213may select the BPP (e.g., the BPP according to Equation 1) associated with the communication environment. The network prediction module1211may identify the packet loss and provide the packet loss to the AI scaling manager1213, and accordingly, the AI scaling manager1213may identify whether the packet loss exceeds the designated threshold loss value. Meanwhile, the above-described selection conditions of the BPP are exemplary and are not limited thereto.

FIG.13Ais a view illustrating image transmission by an electronic device according to an embodiment of the disclosure.

Referring toFIG.13A, according to an embodiment of the disclosure, the electronic device101(e.g., the processor120) may execute a AI scaling manager1301and an AI model1302for down-scaling. The relatively high-resolution image captured by the camera module180may be provided to the AI model1302through the AI scaling manager1301, or may be provided directly from the camera module180to the AI model1302. The AI scaling manager1301may receive the average value of the sizes of a designated number (e.g., K which may be a natural number of 1 or more) of encoded images. The AI scaling manager1301may receive codec parameters including the framerate and the resolution. The AI scaling manager1301may identify, e.g., the BPP (e.g., the BPP as shown in Equation 5) of the value obtained by dividing the average by the resolution. The AI scaling manager1301may provide the image and the BPP to the AI model1302. The AI model1302may receive the image and the BPP as input values and output a low-resolution image. Meanwhile, in the embodiment ofFIG.13A, it has been described that the AI scaling manager1301selects to use the BPP of the value obtained by dividing the average by the resolution, but this is exemplary. For example, the AI scaling manager1301may select to use the BPP of the value obtained by dividing the average by the resolution, based on the number of pre-encoded images being greater than or equal to the designated number K. However, when the number of pre-encoded images is less than the designated number K, the AI scaling manager1301may be configured to use the value obtained by dividing the bitrate as shown in Equation 1 by the product of the framerate and the resolution (e.g., the BPP as shown in Equation 1). Alternatively, the AI scaling manager1301may select to use the BPP of the value obtained by dividing the average by the resolution based on the bitrate not changing rapidly, but when the bitrate changes rapidly, the AI scaling manager1301may be configured to use the value obtained by dividing the bitrate as shown in Equation 1 by the product of the framerate and the resolution (e.g., the BPP as shown in Equation 1).

FIG.13Bis a view illustrating image reception by an electronic device according to an embodiment of the disclosure.

Referring toFIG.13B, according to an embodiment of the disclosure, the electronic device101(e.g., the processor120) may execute an AI scaling manager1321and a AI model1323for up-scaling. The AI model1323may receive an image having a relatively low-resolution. For example, the AI model1323may receive a relatively low-resolution image decoded by the decoder. The AI scaling manager1321may receive the average value of the sizes of a designated number (e.g., K which may be a natural number of 1 or more) of encoded images. For example, the decoder may identify the size of the received encoded image (or bitstream) and provide it to the AI scaling manager1321, or may identify the average and provide it to the AI scaling manager1321. It will be understood by one of ordinary skill in the art that when the size of the encoded image (or bitstream) received from the decoder is received, the AI scaling manager1321may be configured to identify the average. The AI scaling manager1321may receive codec parameters including the framerate and the resolution. The AI scaling manager1321may receive the predicted bitrate.

According to an embodiment of the disclosure, the AI scaling manager1321may identify whether the packet loss rate exceeds the threshold loss rate Th. When the packet loss rate is relatively large, there is a possibility that there is a difference between the average for the designated number K used on the transmitting side and the average for the designated number K used on the receiving side. Accordingly, when the packet loss rate exceeds the threshold loss rate Th (yes in1322), the AI scaling manager1321may provide a value (e.g., the BPP according to Equation 1) obtained by dividing the bitrate by the product of the framerate and the resolution as a part of the input values to the AI model1323. When the packet loss rate is less than or equal to the threshold loss rate Th (no in1322), the AI scaling manager1321may provide a value obtained by dividing the average by the resolution (e.g., the BPP according to Equation 5) as a part of the input values to the AI model1323. Accordingly, the AI model1323may receive the image and the BPP as input values, and may provide a high-resolution image corresponding thereto.

According to an embodiment of the disclosure, an electronic device101may comprise memory130, a camera module180, a communication module190, and at least one processor120operatively connected to the memory130, the camera module180and the communication module190. The memory130, when executed by the at least one processor120, may cause the electronic device101to establish a call connection with a network based on the communication module190. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify a first image captured based on the camera module180. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device101. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model. The memory130, when executed by the at least one processor120, may cause the electronic device101to transmit the second image through the call connection based on the communication module190.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to as at least part of identifying the first information associated with the first bitrate corresponding to the first image, identify a first bit per pixel (BPP) obtained by dividing the first bitrate by a product of a first framerate associated with the first image and a resolution associated with the first image, as the first information.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image, identify the first BPP as the first information associated with the first bitrate based on at least one first condition being met.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image, identify a second BPP obtained by dividing an average of sizes of a designated number of pre-encoded images by the resolution, as the first information, based on at least one second condition different from the at least one first condition being met or the at least one first condition being not met.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of transmitting the second image, generate a bitstream by encoding the second image. The memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of transmitting the second image, transmit the bitstream through the call connection.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to identify the communication environment based on at least one of a one-way delay, a perceived bitrate, a packet loss rate, or a bandwidth.

According to an embodiment of the disclosure, the artificial intelligence model for down-scaling may include a first portion extracting a feature of the first image, a second portion extracting a feature of the first information, a multiplier cross-multiplying the feature of the first image and the feature of the first information, a third portion for enhancing a result of the cross-multiplying by the multiplier and configuring a residual image, a fourth portion for down-scaling the first image, and an adder for adding an output result of the third portion and an output result of the fourth portion. The result of adding by the adder may be provided as the second image.

According to an embodiment of the disclosure, the artificial intelligence model for down-scaling may be a ResNet. The first portion may include at least one convolution layer. The second portion may be a DenseNet. The third portion may include at least one convolution layer. The fourth portion may be a Bicubic down scaler.

According to an embodiment of the disclosure, a method for operating an electronic device101may comprise identifying a first image captured based on a camera module180of the electronic device101. The method for operating the electronic device101may comprise identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device101. The method for operating the electronic device101may comprise identifying a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model. The method for operating the electronic device101may comprise transmitting the second image through the call connection based on a communication module190of the electronic device101.

According to an embodiment of the disclosure, in a storage medium storing at least one computer-readable instruction, the at least one instruction may, when executed by at least one processor120of an electronic device101, enable the electronic device101to perform at least one operation. The at least one operation may include identifying a first image captured based on a camera module180of the electronic device101. The at least one operation may include identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device101. The at least one operation may include identifying a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model. The at least one operation may include transmitting the second image through the call connection based on a communication module190of the electronic device101.

According to an embodiment of the disclosure, an electronic device101may comprise memory130, a display module, a communication module190, and at least one processor120operatively connected to the memory130, the display module and the communication module190. The memory130, when executed by the at least one processor120, may cause the electronic device101to establish a call connection with a network based on the communication module190. The memory130, when executed by the at least one processor120, may cause the electronic device101to receive the first image through the call connection based on the communication module190. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device101. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify a second image corresponding to the first image output from an artificial intelligence model for up-scaling, trained to receive information associated with a low-resolution image and a bitrate as an input value to output a high-resolution image, by inputting the first image and the first information to the artificial intelligence model. The memory130, when executed by the at least one processor120, may cause the electronic device101to control the display module to display at least a portion of the second image.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image, identify a first bit per pixel (BPP) obtained by dividing the first bitrate by a product of a first frame rate associated with the first image and a resolution associated with the first image, as the first information.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image, identify the first BPP as the first information associated with the first bitrate based on at least one first condition being met.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image, identify a second BPP obtained by dividing an average of sizes of a designated number of pre-encoded images by the resolution, as the first information, based on at least one second condition different from the at least one first condition being met or the at least one first condition being not met.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of receiving the first image, receive a bitstream through the call connection. The memory130, when executed by the at least one processor120, may cause the electronic device101to, as at least part of receiving the first image, identify the first image by decoding the bitstream.

According to an embodiment of the disclosure, the memory130, when executed by the at least one processor120, may cause the electronic device101to identify the communication environment based on at least one of a one way delay, a perceived bitrate, a packet loss rate, or a bandwidth.

According to an embodiment of the disclosure, the artificial intelligence model for up-scaling may include a first portion extracting a feature of the first image, a second portion extracting a feature of the first information, a multiplier cross-multiplying the feature of the first image and the feature of the first information, a third portion for enhancing a result of the cross-multiplying by the multiplier and configuring a residual image, a fourth portion for up-scaling the first image, and an adder for adding an output result of the third portion and an output result of the fourth portion. The result of adding by the adder may be provided as the second image.

According to an embodiment of the disclosure, the artificial intelligence model for down-scaling may be a ResNet. The first portion may include at least one convolution layer. The second portion may be a DenseNet. The third portion may include at least one convolution layer. The fourth portion may be a Bicubic down scaler.

According to an embodiment of the disclosure, a method for operating an electronic device101may comprise establishing a call connection with a network based on the communication module190. The method for operating the electronic device101may comprise receiving a first image through the call connection based on a communication module190of the electronic device101. The method for operating the electronic device101may comprise identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device101. The method for operating the electronic device101may comprise identifying a second image corresponding to the first image output from an artificial intelligence model for up-scaling, trained to receive information associated with a low-resolution image and a bitrate as an input value to output a high-resolution image, by inputting the first image and the first information to the artificial intelligence model. The method for operating the electronic device101may comprise controlling a display module of the electronic device101to display at least a portion of the second image.

According to an embodiment of the disclosure, in a storage medium storing at least one computer-readable instruction, the at least one instruction may, when executed by at least one processor120of an electronic device101, enable the electronic device101to perform at least one operation. The at least one operation may include receiving a first image through the call connection based on a communication module190of the electronic device101. The at least one operation may include identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device101. The at least one operation may include identifying a second image corresponding to the first image output from an artificial intelligence model for up-scaling, trained to receive information associated with a low-resolution image and a bitrate as an input value to output a high-resolution image, by inputting the first image and the first information to the artificial intelligence model. The at least one operation may include controlling a display module of the electronic device101to display at least a portion of the second image.

According to an embodiment of the disclosure, a method for training a first AI model for down-scaling and a second AI model for up-scaling comprises identifying training data including a first image which is a high-resolution image and first information associated with a bitrate. The training method may comprise identifying a second image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model. The training method may comprise identifying a third image, which is a high-resolution image, output from the second AI model, based on inputting the second image and the first information to the second AI model. The training method may comprise identifying a fourth image by down-scaling the first image. The training method may comprise identifying a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image. The training method may comprise training at least a portion of the first AI model and the second AI model based on the total loss.

According to an embodiment of the disclosure, the first information associated with the bitstream may be a bit per pixel (BPP) obtained by dividing the bitrate by a product of a first framerate associated with the first image and a resolution associated with the first image.

According to an embodiment of the disclosure, the training method may comprise identifying a fifth image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model. The training method may further comprise identifying a sixth image by encoding the fifth image and decoding a result of the encoding. The training method may comprise identifying a seventh image, which is a high-resolution image, output from the second AI model, based on inputting the sixth image and the first information to the second AI model. The training method may further comprise identifying an eighth image obtained by enhancing the first image. The training method may further comprise identifying a total loss based on the seventh image and the eighth image. The training method may further comprise training the second AI model based on the total loss.

According to an embodiment of the disclosure, a second loss corresponding to the second image and the fourth image may be a loss between images obtained by enhancing the second image and the fourth image.

According to an embodiment of the disclosure, an electronic device101for training a first AI model for down-scaling and a second AI model for up-scaling comprises memory130and at least one processor120. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify training data including a first image, which is a high-resolution image, and first information associated with a bitrate. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify a second image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify a third image, which is a high-resolution image, output from the second AI model, based on inputting the second image and the first information to the second AI model. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify a fourth image by down-scaling the first image. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify a fifth image by enhancing the fourth image. The memory130, when executed by the at least one processor120, may cause the electronic device101to identify a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fifth image. The memory130, when executed by the at least one processor120, may cause the electronic device101to train at least a portion of the first AI model and the second AI model based on the total loss.

According to an embodiment of the disclosure, in a storage medium storing at least one computer-readable instruction, the at least one instruction may, when executed by at least one processor120of an electronic device101, enable the electronic device101to perform at least one operation. The at least one operation may include identifying training data including a first image, which is a high-resolution image, and first information associated with a bitrate. The at least one operation may include identifying a second image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model for down-scaling. The at least one operation may include identifying a third image, which is a high-resolution image, output from the second AI model, based on inputting the second image and the first information to the second AI model for up-scaling. The at least one operation may include identifying a fourth image by down-scaling the first image. The at least one operation may include identifying a fifth image by enhancing the fourth image. The at least one operation may include identifying a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fifth image. The at least one operation may include training at least a portion of the first AI model and the second AI model based on the total loss.