Patent Description:
In recent years, with rapid development of the electronic industry and communications technologies, new services on basis of data, voice, and videos develop rapidly. Fast development of microelectronics technologies and computer software and hardware technologies lays a foundation for increasingly complex processing of image processing devices and makes personalization of the image processing devices possible, so that a terminal is no longer restricted by a network to some extent and may have increasingly powerful functions. In addition, users also have urgent requirements for terminals, and expect the terminals to have more powerful, more flexible, and more convenient functions. Development of information technologies enables intelligent, mobile, and multi-functional terminal technologies.

As mobile terminals are increasingly popular, particularly, as smartphones develop progressively, people's life becomes more convenient and people can enjoy achievements brought by advanced technologies. Intelligent mobile terminals such as smartphones are accepted by more people owing to many advantages such as powerful operating systems, large-capacity storage space, and convenience in installing various software. In comparison with conventional mobile terminals, the intelligent mobile terminals can install more third-party applications. In an intelligent mobile terminal having an Android (Android) system, the operating system generally sequentially includes an application layer, a framework layer, a runtime, a core class library, a hardware abstraction layer, and a Linux kernel layer. Generally, each Android application developer develops applications based on implementation of core functions of the Android system, including the framework layer, the core class library, and the like. The application layer of the Android system consists of all applications running on the Android device. The application layer not only includes system applications (pre-installed on the intelligent mobile terminal along with the Android system) such as a call application, a short message application, a contacts application, and the like, but also includes other third-party applications subsequently installed on the device. The third-party applications are developed based on a software development kit (software development kit, SDK) provided by Android and are restricted by an SDK interface. The system applications pre-installed on the device may call interfaces and modules of the entire framework layer. In an operating system of an existing intelligent mobile terminal, after being installed and obtaining a system grant, a third-party application can directly call a control interface from an Android framework layer. For example, during image display, an application program usually calls an ImageView control at the framework layer of the Android system to draw an image. However, to reduce traffic during the image display, many image details are compressed in consideration of frequency bandwidth, resulting in a low image resolution. <CIT> discusses example processing methods and devices. In one example method, resolution of each layer of multiple layers of an application program are determined. Resolution of a first layer is lower than resolution of another layer. The multiple layers are created according to the resolution of each layer. Rendering processing is performed on the multiple layers. Scaling processing is performed on a second layer obtained after the rendering processing. The determined resolution of the second layer is different from resolution of a preset area displaying the second layer on a display screen. Resolution of the second layer obtained after the scaling processing is the same as the resolution of the preset area. A layer other than the second layer obtained after the rendering processing is composited with the second layer obtained after the scaling processing. "<NPL> Et Al discusses an improvement of fast curvature based interpolation in image super resolution is proposed by using genetic algorithm optimization. The fast curvature based image super resolution requires low computation time, and it is suitable for real time video superresolution. "https://mkyong. com/android/android-imageview-example/" discusses using ImageView to display a "png" image. "Testing for Poor Responsiveness in Android Applications" by Yang Shenqian Et A1 discusses a systematic technique to uncover and quantify common causes of poor responsiveness in Android software. When test cases are executed against the application GUI, artificial long delays are inserted at typical problematic operations (e.g., at calls that access the network). This test amplification approach may exhibit increased response times for GUI events, which demonstrates the effects of expensive operations on poor responsiveness observed by the user. The proposed approach successfully uncovered <NUM> responsiveness problems in eight open-source Android applications, due to inappropriate usage of resources such as network, flash storage, on-device database, and bitmaps.

This application provides an image processing method and device, to resolve the following problem: A resolution is low when an application program in an existing terminal device displays image information. The scope of protection of the present invention is set out by the appended claims.

According to a first aspect, an embodiment of this application provides an image processing method. The method is applicable to an image processing device having an operating system, including: receiving, by an image processing module of the operating system, an instruction of a first application program to call the image processing module of the operating system. Because the instruction carries a to-be-displayed image, the image processing module performs image optimization processing on the to-be-displayed image and displays an image obtained after the image optimization processing.

According to the foregoing method, the image processing module of the operating system of the image processing device is improved, and an image optimization function is added, to perform image optimization processing on to-be-displayed images in different application programs at an application layer. To be specific, when a multimedia file in an application program calls an interface of the image processing module of the operating system to display an image, an image optimization processing process of the image processing module is first performed, and an optimized image is eventually displayed. For example, the optimized image has an increased resolution and has higher definition.

In a possible design, the operating system is an Android operating system and the image processing module is an ImageView class at a framework layer of the Android operating system. In this way, the ImageView class can obtain a width and a height of the to-be-displayed image from the instruction, and then the ImageView class determines that the width and the height of the to-be-displayed image meet a specified condition. The specified condition is as follows: a difference between a height of a screen of the image processing device and the height of the to-be-displayed image is less than a second threshold; and/or a difference between a width of the screen of the image processing device and the width of the to-be-displayed image is less than a third threshold. In other words, the ImageView class can perform image optimization on only an image of a sufficient size, and this can improve image optimization efficiency.

The operating system is an Android operating system and the image processing module is a BitmapFactory class at a framework layer of the Android operating system. In this case, before the image processing module performs image optimization processing on the to-be-displayed image, the to-be-displayed image is decoded. This method mainly compensates for image optimization in a scenario in which the application program has an ImageView class. Because when the application program at the application layer already has an ImageView class, ImageView at the framework layer is not called, but the BitmapFactory class (BitmapFactory) of the framework layer still is called, to decode the to-be-displayed image and obtain a decoded image. Therefore, the image optimization function can be extended on BitmapFactory. In this case, once an application program at the application layer calls this interface, image optimization is triggered.

When determining that a resolution of the to-be-displayed image is less than a first threshold, the image processing module performs super-resolution processing on the to-be-displayed image. With super-resolution processing, the resolution of the original image can be adjusted, and therefore the optimized image has higher definition.

The image processing module determines, based on an identifier of the first application program in the instruction, that the first application program has a super-resolution processing permission. Apparently, this can improve the image optimization efficiency and facilitate centralized processing of multimedia files having an image element.

The image processing module determines whether the identifier of the first application program exists in a preset whitelist; if the identifier of the first application program exists in the preset whitelist, the image processing module determines that the first application program has an image optimization permission; or if the identifier of the first application program does not exist in the preset whitelist, the image processing module does not perform image optimization on the image of the application program. The whitelist can be updated by a user anytime and therefore is easy to be controlled.

Specifically, a method for performing super-resolution processing on the to-be-displayed image by the image processing module may be: adding, by the image processing module, the to-be-displayed image to a task queue as a task object; and
determining, by the image processing module based on the resolution of the to-be-displayed image, an image optimization algorithm corresponding to the to-be-displayed image; and performing, by the image processing module by using the corresponding image optimization algorithm, super-resolution processing on the task object corresponding to the to-be-displayed image in the task queue.

In addition, after determining that the super-resolution processing on the task object corresponding to the to-be-displayed image is completed, the image processing module releases memory space corresponding to the task object, to facilitate memory management.

In addition, a first task in the task queue is allocated to a first processor to perform super-resolution processing, and a second task in the task queue is allocated to a second processor to perform super-resolution processing, to implement heterogeneous acceleration.

According to a second aspect, an embodiment of this application provides an image processing apparatus. The image processing apparatus has a function of implementing behaviors of the image processing module in the operating system in the foregoing method embodiment. The function may be implemented by hardware by executing corresponding software. The hardware or software includes one or more modules corresponding to the foregoing function.

The image processing apparatus includes a receiving unit and a processing unit, where.

The processing unit is specifically configured to: when determining that a resolution of the to-be-displayed image is less than a first threshold, perform super-resolution processing on the to-be-displayed image.

The image processing apparatus further includes a determining unit, configured to determine, based on an identifier of the first application program, that the first application program has a super-resolution processing permission.

The determining unit is specifically configured to: determine whether the identifier of the first application program exists in a preset whitelist; and if the identifier of the first application program exists in the preset whitelist, determine that the first application program has an image optimization permission.

In a possible design, the processing unit is specifically configured to:.

Further, the processing unit is further configured to: after determining that the super-resolution processing on the task object corresponding to the to-be-displayed image is completed, release memory space corresponding to the task object.

In addition, in a possible design, a first task in the task queue is allocated to a first processor to perform super-resolution processing, and a second task in the task queue is allocated to a second processor to perform super-resolution processing, to accelerate image processing.

In a possible design, the image processing apparatus is an ImageView class in the operating system, and the instruction further includes a width and a height of the to-be-displayed image. The processing unit is further configured to determine that the width and the height of the to-be-displayed image meet a specified condition, where the specified condition is as follows: a difference between a height of a screen of the terminal device and the height of the to-be-displayed image is less than a second threshold; and/or a difference between a width of the screen of the terminal device and the width of the to-be-displayed image is less than a third threshold.

In a possible design, the image processing apparatus is a BitmapFactory class at a framework layer of an Android operating system. In this case, before performing image optimization processing on the to-be-displayed image, the image processing apparatus decodes the to-be-displayed image.

According to a third aspect, an embodiment of this application provides an image processing device. The image processing device includes a memory, a display, and a processor. The processor may be a central processing unit (central processing unit, CPU), a digital processing unit, or the like. The processor performs an image optimization function based on an instruction of a first application program to call an image processing module of an operating system. The memory is configured to store an instruction of the first application program, a program instruction of the operating system, and a program to be executed by the processor.

The display is configured to display, on a human-computer interaction interface of the first application program, an image obtained after image optimization processing performed by the processor.

Specifically, when the processor is configured to: when determining that a resolution of the to-be-displayed image is less than a first threshold, perform super-resolution processing on the to-be-displayed image.

In a possible design, the instruction of the first application program further includes an identifier of the first application program. The processor is further configured to determine, based on the identifier of the first application program, that the first application program has a super-resolution processing permission.

In a possible design, the processor is specifically configured to: determine whether the identifier of the first application program exists in a preset whitelist; and if the identifier of the first application program exists in the preset whitelist, determine that the first application program has an image optimization permission.

In a possible design, the processor is specifically configured to: add the to-be-displayed image to a task queue as a task object; and determine, based on the resolution of the to-be-displayed image, an image optimization algorithm corresponding to the to-be-displayed image; and
perform, by using the corresponding image optimization algorithm, super-resolution processing on the task object corresponding to the to-be-displayed image in the task queue.

In a possible design, the operating system is an Android operating system, the image processing module is an ImageView class or a BitmapFactory class at a framework layer of the Android operating system, and the processor may optimize an image in the following two manners. Manner <NUM> does not form part of the claimed invention.

Manner <NUM>: The processor determines that a width and a height of the to-be-displayed image meet a specified condition and performs super-resolution processing on an image that meets the condition. The specified condition is as follows: a difference between a height of a screen of the image processing device and the height of the to-be-displayed image is less than a second threshold; and/or a difference between a width of the screen of the image processing device and the width of the to-be-displayed image is less than a third threshold.

Manner <NUM>: The processor decodes the to-be-displayed image and performs super-resolution processing on an image that meets a condition.

In a possible design, the processor is further configured to: after determining that the super-resolution processing on the task object corresponding to the to-be-displayed image is completed, release memory space corresponding to the task object.

In a possible design, a first task in the task queue is allocated to a first processor to perform super-resolution processing, and a second task in the task queue is allocated to a second processor to perform super-resolution processing, to implement heterogeneous acceleration.

According to a fourth aspect, an embodiment of this application further provides a computer storage medium, where the storage medium stores a software program, and when the software program is read and executed by one or more processors, the method provided in any design of the first aspect can be implemented.

According to a fifth aspect, an embodiment of this application further provides a computer program product, and when the computer program product is executed by a computer, the computer is enabled to execute the method provided in any design of the first aspect.

In the solutions provided in the embodiments of this application, the image processing module in the operating system is improved and it is unnecessary to develop each application program separately. Therefore, the solutions are highly reusable. In addition, when image information of multimedia files from different application programs is displayed, automatic image optimization is implemented, thereby improving user experience.

The following further describes in detail embodiments of this application with reference to accompanying drawings.

An image processing method provided in an embodiment of the present invention is applicable to an image processing device shown in <FIG>. The image processing device includes an application layer <NUM> and a framework layer <NUM>. The application layer <NUM> includes a first application program <NUM> and a plurality of other application programs, and the framework layer <NUM> includes an image processing module <NUM>. The image processing module <NUM> receives an image transmitted by the first application program <NUM> and is configured to perform image optimization processing on the image. The image processed by the image processing module <NUM> is transmitted to the first application program <NUM>, so that the first application program <NUM> displays the processed image. It can be understood that the framework layer <NUM> further includes a storage module, configured to buffer a to-be-processed image and a processed image. The following image processing process is used as an example to describe a workflow of the image processing module.

After receiving a to-be-displayed image, the image processing module <NUM> first obtains a resolution of the to-be-displayed image and determines whether the resolution is less than a threshold, and if the resolution is less than the threshold, performs super-resolution processing on the to-be-displayed image and displays an image obtained after the super-resolution processing. The so-called super-resolution processing means increasing a resolution of an original image by using a hardware or software method. A process of obtaining a high-resolution image through a series of low-resolution images is super-resolution reconstruction. The image processing module <NUM> returns the image obtained after the super-resolution processing to the first application program <NUM> for display on an interface.

An image processing method provided in this embodiment of the present invention is also applicable to a mobile phone shown in <FIG>. The following first briefly describes a specific structural composition of the mobile phone.

<FIG> is a schematic structural diagram of hardware of a mobile phone applied to an embodiment of this application. As shown in <FIG>, a mobile phone <NUM> includes a display device <NUM>, a processor <NUM>, and a memory <NUM>. The memory <NUM> may be configured to store a software program and data, and the processor <NUM> runs the software program and the data that are stored in the memory <NUM>, to execute various function applications of the mobile phone <NUM> and perform data processing. The memory <NUM> may mainly include a program storage area and a data storage area. The program storage area may store an operating system, an application program required by at least one function (such as an image capture function), and the like; and the data storage area may store data (such as audio data, a phone book, and an image) created based on use of the mobile phone <NUM>, and the like. In addition, the memory <NUM> may include a high-speed random access memory, and may further include a nonvolatile memory, for example, at least one magnetic disk storage device, a flash memory device, or another volatile solid-state storage device. The processor <NUM> is a control center of the mobile phone <NUM>, and is connected to all parts of the entire mobile phone by using various interfaces and lines. The processor <NUM> runs or executes the software program and/or the data that are/is stored in the memory <NUM>, to perform various functions of the mobile phone <NUM> and perform data processing, thereby performing overall monitoring on the mobile phone. The processor <NUM> may include one or more general purpose processors, may further include one or more DSPs (digital signal processor, digital signal processor), and may also include one or more ISPs (image signal processor, image signal processor), and is configured to perform related operations, to implement the technical solutions provided in the embodiments of this application.

The mobile phone <NUM> further includes a camera <NUM> for capturing an image or shooting a video. The camera <NUM> may be a common camera, or may be a focus camera.

The mobile phone <NUM> may further include an input device <NUM>, configured to receive digital information, character information, or a contact touch operation/non-contact gesture that is input, and generate signal input that is related to user settings and function control of the mobile phone <NUM>, and the like.

The display device <NUM> includes a display panel <NUM>, configured to display information that is input by a user, information provided for a user, various menu interfaces of the mobile phone <NUM>, and the like, and is mainly configured to display a to-be-detected image obtained by a camera or a sensor in the mobile phone <NUM> in this embodiment of this application. Optionally, the display panel <NUM> may be configured in a form of a liquid crystal display (liquid crystal display, LCD), an OLED (organic light-emitting diode, organic light-emitting diode), or the like.

In addition to the foregoing parts, the mobile phone <NUM> may further include a power supply <NUM>, configured to supply power to other modules. The mobile phone <NUM> may further include one or more sensors <NUM>, such as an image sensor, an infrared sensor, and a laser sensor. The mobile phone <NUM> may further include a radio frequency (radio frequency, RF) circuit <NUM>, configured to perform network communication with a wireless network device, and may further include a Wi-Fi module <NUM>, configured to perform Wi-Fi communication with another device to obtain images, data, or the like transmitted by the another device.

Based on the foregoing description, the embodiments of this application provide an image processing method and an image processing device, to resolve a problem of unclear image display of a multimedia file in an application program. In this application, the method and the image processing device are based on a same inventive concept. Because the method and the image processing device have similar problem-resolving principles, reference may be mutually made to implementations of the image processing device and the method. No repeated description is provided.

In the embodiments of this application, the image processing module of the operating system of the image processing device is mainly improved and an image optimization function is added, to perform image optimization processing on to-be-displayed images in different application programs at an application layer. That is, in the embodiments of this application, layers (such as a framework layer) in the operating system other than the application layer are improved, an image optimization program is added, and related hardware is called to implement optimization processing. When a multimedia file in the application program calls the image processing module of the operating system for image display, an image optimization processing process of the image processing module is first performed, and an optimized image is eventually displayed. In this case, a resolution is increased, and higher definition is also achieved. In the embodiments of this application, the image processing module in the operating system is improved and it is unnecessary to develop each application program separately. A user operating an application program at the application layer is unaware of the entire image optimization process. Therefore, the method is highly reusable and more automated.

In the following, some terms in this application are described, to help a person skilled in the art has a better understanding.

With reference to the operating system architecture of the image processing device in <FIG>, a specific process of an image processing method is described in detail in the following embodiments of this application. Referring to <FIG>, a specific procedure of the method may include the following steps.

A first application program sends an instruction to an image processing module of an operating system to call the image processing module to perform image optimization.

Specifically, when detecting that a user is viewing an image, the first application program sends the instruction to the image processing module. For example, the user receives an image sent by a friend when using WeChat. In a dialog interface of WeChat, the received image is displayed in a thumbnail. When the user taps the thumbnail to view a large version of the image, WeChat calls the image processing module.

When determining that a resolution of the to-be-displayed example, the user receives an image sent by a friend when using WeChat. In a dialog interface of WeChat, the received image is displayed in a thumbnail. When the user taps the thumbnail to view a large version of the image, WeChat calls the image processing module.

When determining that a resolution of the to-be-displayed image is less than a first threshold, the image processing module performs super-resolution processing on the to-be-displayed image and displays an image obtained after the super-resolution processing.

In step 202a, because the instruction carries the to-be-displayed image and the resolution of the to-be-displayed image, the image processing module first obtains the resolution of the to-be-displayed image and then perform determining on the resolution. If determining that the resolution is less than the first threshold, the image processing module starts super-resolution processing; or if determining that the resolution is not less than the first threshold, the image processing module does not perform super-resolution processing.

It should be noted that, in addition to the image optimization method of super-resolution processing, another image optimization method may further be used to optimize an image, for example, adjust brightness, saturation, or a color of the image, beautifying a human face, or the like. Alternatively, these image optimization methods may be used together, or different image optimization methods may be triggered based on different detection conditions. For example, when it is identified that a current mode is a night reading mode, brightness of an image is reduced; or when it is identified that an image includes a human face, an image optimization method for beautifying the human face is performed, or the like.

In consideration that image optimization processing aims to achieve an image quality improvement, and such an improvement has different impact and significance for users in different applications. For example, a user of a microblog usually browses short videos or photos, and therefore this type of application program such as the microblog has a relatively high requirement for an image resolution. For another example, in a mailbox application, an email includes mostly text information, and usually has only a small amount of image information or no image information inserted, and therefore this type of application program has a relatively low requirement for an image resolution. If consideration is performed based on frequency bandwidth as in a conventional method, some pixels are lost, which reduces an image resolution and severely affects user experience. Therefore, the image processing module determines, based on an identifier of the first application program carried in the instruction, whether the first application program has an image optimization permission, and performs image optimization on an application program that has the image optimization permission.

That is, the image processing module prestores a whitelist having the image optimization permission and the whitelist includes identifiers of application programs at the application layer. If the image processing module determines that the identifier of the application program obtained in the instruction is in the whitelist, the application program apparently has the image optimization permission, and super-resolution processing may be further triggered; or if the image processing module determines that the identifier of the application program obtained in the instruction is not in the whitelist, super-resolution processing is not performed. Alternatively, in an embodiment that is not part of the claimed invention, the image processing module authenticates the identifier of the application program. If the authentication succeeds, it is proved that the application program has the image optimization permission, and super-resolution processing may be further triggered. If the authentication fails, super-resolution processing is not performed.

In another possible design, to improve image processing efficiency, before super-resolution processing is performed, another condition may further be used to further determine whether the to-be-displayed image needs image optimization. For example, after obtaining an image, the image processing module determines whether super-resolution processing has been performed on the image. If super-resolution processing has been performed on the image, super-resolution processing is not performed on the image again.

In consideration that when many to-be-displayed images exist in an application program, the image processing module encounters relatively high performance and memory load. Therefore some images may be filtered out, for example, an image with a size meeting a preset condition is processed. For example, the image processing module determines whether an image size of the to-be-displayed image is close to that of a full screen. Super-resolution processing is performed only when the image size is close to that of the screen. As shown in <FIG>, a width of the to-be-displayed image is consistent with a width of the screen, and therefore the to-be-displayed image meets the condition for performing super-resolution processing. In actual operations, image widths of some applications are slightly less than the width of the screen. Therefore, when a difference between a height of the screen and a height of the to-be-displayed image is less than a second threshold, or when a difference between the width of the screen of the image processing device and the width of the to-be-displayed image is less than a third threshold, that is, when either of the two conditions is met, super-resolution processing is performed on the to-be-displayed image. The second threshold and the third threshold may be same. For example, when the width of the to-be-displayed image meets [<NUM>, <NUM>] times the width of the screen, or when the height of the to-be-displayed image meets [<NUM>, <NUM>] times the height of the screen, super-resolution processing may still be performed.

At present, a common operating system used by the image processing device is an Android operating system. An embodiment of this application further describes the foregoing image processing method with reference to a system architecture of the Android operating system. From a high layer to a low layer, the Android operating system generally sequentially includes: an application layer, a framework layer, a runtime, a core class library, a hardware abstraction layer, and a Linux kernel layer. The image processing module in this embodiment of this application belongs to a functional module at the framework layer.

As shown in <FIG>, an Android operating system architecture having an image optimization function includes an application layer <NUM>, a framework layer <NUM>, a hardware abstraction layer <NUM>, and a kernel chip <NUM>. A scene recognition module <NUM> in an image processing module at the framework layer is configured with various conditions for determining whether image optimization is needed, for example, determining whether a first application program that calls the image processing module exists in a whitelist for image optimization, whether a to-be-displayed image needs to be displayed in full screen, whether a width or a height of the to-be-displayed image is close to a width or a height of a screen, whether a resolution of the to-be-displayed image meets a resolution threshold, whether the to-be-displayed image includes a human face, or the like. An ImageView class (ImageView) <NUM> and a BitmapFactory class (BitmapFactory) <NUM> are configured to perform image optimization on the to-be-displayed image, and may specifically manage a task queue and memory calling for image optimization. An image optimization algorithm in a HiAI service is called to perform optimization processing the image. An HiAI service platform <NUM> at the framework layer includes various image optimization algorithms, such as a DNN hard algorithm, a RAISR soft algorithm, and the like. The DNN hard algorithm is a super-resolution image hard algorithm, and performance and an effect are improved through interaction with a neural network processor (IPU). The RAISR soft algorithm is a super-resolution image soft algorithm and is mainly specific to low- and mid-range image processing devices without IPU hardware. In addition, the framework layer further includes a storage module <NUM>, whose function is already illustrated in <FIG> and is not described herein in detail again.

Application programs at the application layer are different. Some application programs may call the ImageView class (ImageView) <NUM> to draw an image, while some application programs may not call ImageView but call the BitmapFactory class (BitmapFactory) <NUM>. Therefore, the image processing module specifically implements image optimization mainly in the following two manners. Manner <NUM> is not part of the claimed invention.

Manner <NUM>: Most application programs at the application layer may call the ImageView class <NUM> to display an image, and therefore a capacity of the image optimization function can be expanded on the ImageView class. In this case, the application programs at the application layer may call the ImageView class to trigger the foregoing image processing method. As shown in <FIG>, when different application programs at an application layer <NUM> display images, ImageView classes are created separately, for example, ImageView <NUM> of a first application program, ImageView <NUM> of a second application program, and ImageView <NUM> of a third application program. An image processing module <NUM> at a framework layer <NUM> is further triggered to call a heterogeneous processor <NUM> to perform image optimization and an ImageView class <NUM> in the image processing module <NUM> is further triggered to call an image optimization algorithm to perform image optimization processing. In a process of performing image optimization processing, the image processing module performs heterogeneous acceleration by using a plurality of processors, to accelerate the image optimization process.

Manner <NUM>: When an application program at the application layer <NUM> already has the ImageView class <NUM>, the application program does not call ImageView at the framework layer <NUM>, but still calls a BitmapFactory class (BitmapFactory) <NUM> at the framework layer, to decode a to-be-displayed image and obtain a decoded image. Therefore, the image optimization function can be extended on BitmapFactory. In this case, once the application program at the application layer calls the BitmapFactory class, the image processing method is triggered. As shown in <FIG>, in step <NUM>, in an image display process, a first application program at an application layer calls BitmapFactory at a framework layer of an operating system to decode a multimedia file; in step <NUM>, a scene recognition module performs a series of condition-based determining on the decoded image, for example, identification of a resolution value, and an image that meets a condition is added to an image optimization queue; in step <NUM>, in consideration that some images such as the first few images or the last few images usually include unimportant information, image optimization on these images may be directly skipped; and in step <NUM>, after filtering, super-resolution processing is sequentially performed on remaining images.

For Manner <NUM>, specifically, in an Android operating system, a most commonly used image display module is ImageView (image view module). According to analysis of Android ImageView source code, regardless of which manner is used to create ImageView of each application program, all ImageView call initImageView() (an initial image view interface) to perform general data initialization. Therefore, in this embodiment of this application, a condition-based determining process related to an image optimization permission is added to this interface, to determine whether a current application program has the image optimization permission. The following uses supper-resolution processing as an example for description, as shown in <FIG>.

Step <NUM>. An application program creates ImageView to call an initImageView() interface to perform data initialization.

Step <NUM>. A plurality of conditions are set in initImageView() to determine whether the current application program has an image optimization capability permission, where the conditions are as follows:.

If the foregoing three conditions are met, it can be determined that ImageView currently created by the application program has the image optimization permission, and subsequent determining logic may be continued, that is, step <NUM> may be performed; otherwise, image optimization processing is not continued, that is, step <NUM> is performed to draw an image for display.

Because image optimization processing aims to achieve an image quality improvement, and such an improvement has different impact and significance for users in different applications and needs to be differentiated based on importance and a function value. Therefore, at present, image optimization processing is enabled only for some specific applications. In addition, in Android applications, a package (package) name usually corresponds to an application program one by one. Therefore, a package name of the current application program may be obtained by calling getPackageName(). If the package name of the application program is in the whitelist, it is determined that ImageView allows image optimization, and subsequent scene recognition logic is continued; or if the package name of the application program is not in the whitelist, subsequent behavior of ImageView is with the same as behavior performed without an image optimization feature.

Step <NUM>. An image size is obtained. To continue to determine, based on the image size, whether the image needs image optimization, getIntrinsicWidth() and getIntrinsicHeight() methods of Drawable in an Android View drawing system may be used to obtain a width and a height of an image, so that a width and a height of a to-be-displayed image can be measured by using these methods. In an example of BitmapDrawable, a width and a height of a held bitmap are actually returned.

Step <NUM>. When a difference between a height of the screen and the height of the to-be-displayed image is less than a first threshold, or when a difference between a width of the screen of the image processing device and the width of the to-be-displayed image is less than a second threshold, that is, when either of the two conditions is met, super-resolution processing is performed on the to-be-displayed image, that is, step <NUM> is performed; otherwise, step <NUM> is performed. For example, when the width of the to-be-displayed image meets [<NUM>, <NUM>] times the width of the screen, or when the height of the to-be-displayed image meets [<NUM>, <NUM>] times the height of the screen, it can be determined that a size of the to-be-displayed image is large enough, and therefore image optimization is necessary for the image.

Step <NUM>. ImageView draws specified Drawable data by using a standard method onDraw() of View. Content is drawn by using the standard method draw() of Drawable. For example, BitmapDrawable is to draw content of a bitmap on ImageView, and ColorDrawable is to paint color on ImageView.

Step <NUM>. It is determined whether a status of a super-resolution optimization algorithm device is normal. When images are displayed in batches, a task queue of the super-resolution optimization algorithm device is congested. Therefore, it is required to determine whether the status of the super-resolution optimization algorithm device is normal. If the status of the super-resolution optimization algorithm device is normal, step <NUM> is performed; if the status of the super-resolution optimization algorithm device is abnormal, step <NUM> is performed.

Step <NUM>. In consideration of performance, a memory, and effects, image optimization processing is performed only on images with resolutions in a particular range. Different application programs have different requirements on a resolution value setting rule. For example, a microblog has a higher image resolution requirement than other applications. Image optimization is triggered only when the resolution of the application program is less than the corresponding value setting rule. Therefore, it is required to further determine whether the resolution meets the specified condition. If the resolution meets the specified condition, step <NUM> is performed; or if the resolution does not meet the specified condition, step <NUM> is performed.

Step <NUM>. Super-resolution processing is performed on a to-be-displayed image that meets the foregoing condition. The super-resolution algorithm executed for the super resolution is corresponding to an image resolution value setting of the application program. Afterwards, step <NUM> is repeated, to re-draw and display an optimized image. It should be noted that, after step <NUM> is performed, the image optimization procedure ends.

Step <NUM>. Though ImageView does not perform image optimization, ImageView still needs to draw specified Drawable data by using a standard method onDraw() of View. Content is drawn by using the standard method draw() of Drawable, as shown in step <NUM>.

It should be noted that, there is no strict sequence for the foregoing determining. Generally, in this embodiment of this application, it is first determined whether a switch is on, then an image size is measured to determine whether the image size is large enough, and it is further determined whether the resolution is less than the specified standard.

<FIG> shows a framework and a procedure for implementing image optimization by using an ImageView class. An operating system includes an application layer A10, a framework A20 and an HAL layer/hardware A30. The application layer A10 has an application program A11 that uses ImageView. An image optimization triggering module in ImageView A21 at the framework layer A20 is configured to determine timing for image optimization. A scene recognition module A211 is configured to determine which image needs image optimization processing, for example, determine whether an image size meets a preset condition, whether an image resolution meets a resolution threshold, whether an image includes a human face. Specifically, the foregoing steps <NUM> to <NUM> and steps <NUM> to <NUM> are included. In practice, different conditions may be set based on image optimization purposes, to determine which image needs image optimization processing. An optimization task allocation and content management module A212 is configured to manage a multi-threaded task and content for image optimization processing, and may be managed by specifically referring to a manner in <FIG>. An optimization algorithm A22 provides a plurality of image optimization models A221, that is, image optimization algorithms. A heterogeneous optimization processing module A222 is configured to call different processors A32 to accelerate image optimization processing. The HAL layer/hardware is configured to implement display and implement hardware driving of a processor. The application A11 sends an image to ImageView A21. The image optimization triggering module in ImageView A21 determines the timing for image optimization processing. The scene recognition module A <NUM> determines whether the image needs optimization. The image that needs optimization is put into an optimization task queue. The optimization task allocation and memory management module A212 manages optimization tasks. The optimization algorithm A22 is called to perform optimization processing on images in the optimization task queue. During optimization processing implementation, different processors A32 are called by using the heterogeneous optimization processing module A222 to run the optimization algorithm. An optimized image is used to update the image for display on a display A31.

For Manner <NUM>, specifically, a prediction model is provided for image optimization processing in BitmapFactory. The prediction model predicts that super-resolution processing needs to be performed on an image in a display interface, and an image outside the display interface needs to be filtered out. This ensures that image optimization processing can be performed on an image that meets a condition, thereby meeting an end-to-end performance requirement. Specific steps are shown in <FIG>.

Step <NUM>. An image of an application program decodes a to-be-displayed image by calling BitmapFactory. decodeFile() of BitmapFactory, and outputs a source bitmap (source bitmap) object, for example, decodes a to-be-displayed image in JPEG format into a bitmap.

Step <NUM>. BitmapFactory calls a BitmapFactory class of a framework layer to import the source bitmap object.

Step <NUM>. An image processing module internally creates/manages a destination bitmap (destination bitmap), and uses a source bitmap and the destination bitmap of the image as parameters to call a native API of a Hisi DDK, to perform data processing.

Step <NUM>. The Hisi DDK uses an internal binder interface to transmit the source bitmap and the destination bitmap to a HiAI service process through inter-process communication (inter-process communication, IPC), to call an IPU for super-resolution algorithm processing.

Step <NUM>. A super-resolution image processed by the HiAI service process is stored in the destination bitmap, and then is asynchronously returned to a binder process of a graphics library process by using a binder interface.

Step <NUM>. After receiving the image processed by the image processing module, the binder process transmits the destination bitmap to a view display interface for redrawing.

Step <NUM>. The redrawn image is displayed on a display interface of the application program.

Because a bitmap is created in the foregoing image optimization processing process, the bitmap is a factor that is most likely to cause a memory exception in most application scenarios. The bitmap complies with a Java GC mechanism. When no strong reference points to a bitmap object, the bitmap object is released. If an improper pointer holds the bitmap, memory occupation or leakage is caused. Therefore, this embodiment of this application explicitly provides a lifecycle of the bitmap in image optimization processing, for ease of memory management. In addition, there are currently numerous image optimization algorithms, and running and computation requirements are high. Different heterogeneous processors such as a GPU and an FPGA may be further managed, and tasks may be allocated to the heterogeneous processors, to accelerate image optimization processing performance.

For example, it is assumed that a first application program is WeChat, a user A sends a selfie to a user B by using WeChat, and a thumbnail of the selfie is displayed in a WeChat dialog box of the user B. When the user B taps the thumbnail to view the selfie, the application program WeChat in a mobile phone of the user B triggers an image processing module of an operating system to perform an image display process. Therefore, ImageView or BitmapFactory in the image processing module performs image optimization on the selfie, and therefore the selfie viewed by the user B has a higher a resolution and higher definition.

In conclusion, according to the foregoing image processing method, the image optimization function does not need to be repeatedly developed for each third-party application program. Super-resolution processing can be triggered by directly calling an existing interface in the operating system. The method is universal.

In addition, in an embodiment of this application, a control policy is further added for an image optimization queue of an image processing module. As shown in <FIG>, in step <NUM>, the control policy is mainly as follows: After a request, for a super-resolution task, initiated by the image processing module is received by a task queue, the task is added to the queue; in step <NUM>, if a request, for canceling a super-resolution task, initiated by the image processing module is received, the task is deleted from the queue; and in step <NUM>, when another task queue detects that a super-resolution state machine is in a ready state, a task is removed from the task queue and sent to a DDK for processing.

In addition, in consideration that a bitmap is created in a complete image optimization processing process, the bitmap is a factor that is most likely to cause a memory exception in most application scenarios. The bitmap complies with a Java GC mechanism. When no strong reference points to a bitmap object, the bitmap object is released. If an improper pointer holds the bitmap, memory occupation or leakage is caused. Therefore, this embodiment of this application clarifies a lifecycle of the bitmap in image optimization processing, for ease of memory management.

Based on a same inventive concept of the method embodiment, an embodiment of this application provides an image processing apparatus <NUM>. The image processing apparatus belongs to a framework layer of an operating system of an image processing device, and is specifically configured to implement the method described in the embodiment shown in <FIG>. As shown in <FIG>, a structure of the apparatus includes a receiving unit <NUM> and a processing unit <NUM>.

The receiving unit <NUM> is configured to receive an instruction of a first application program to call an image processing module of an operating system, where the instruction carries a to-be-displayed image.

The processing unit <NUM> is configured to perform image optimization processing on the to-be-displayed image and display an image obtained after the image optimization processing.

The processing unit <NUM> is specifically configured to: when determining that a resolution of the to-be-displayed image is less than a first threshold, perform super-resolution processing on the to-be-displayed image.

The image processing apparatus further includes a determining unit <NUM>, configured to determine, based on an identifier of the first application program, that the first application program has a super-resolution processing permission.

The determining unit <NUM> is specifically configured to determine whether the identifier of the first application program exists in a preset whitelist; and if the identifier of the first application program exists in the preset whitelist, determine that the first application program has an image optimization permission.

In a possible design, the processing unit <NUM> is specifically configured to:.

Further, the processing unit <NUM> is further configured to: after determining that the super-resolution processing on the task object corresponding to the to-be-displayed image is completed, release memory space corresponding to the task object.

In a possible design, it is assumed that the image processing apparatus is an ImageView class in the operating system, and the instruction further includes a width and a height of the to-be-displayed image. The processing unit <NUM> is further configured to determine that the width and the height of the to-be-displayed image meet a specified condition, where the specified condition is as follows: a difference between a height of a screen of the terminal device and the height of the to-be-displayed image is less than a second threshold; and/or a difference between a width of the screen of the terminal device and the width of the to-be-displayed image is less than a third threshold.

According to the foregoing embodiments, an embodiment of this application further provides an image processing device, and the image processing device is configured to implement the method described in the embodiment in <FIG>. As shown in <FIG>, the device includes a processor <NUM>, a memory <NUM>, and a display <NUM>.

The processor <NUM> may be a central processing unit (central processing unit, CPU), a digital processing unit, or another optimization function.

The memory <NUM> is configured to store an instruction of a first application program or a program instruction of an operating system.

The display <NUM> is configured to display, on a human-computer interaction interface of the first application program, an image obtained after image optimization processing performed by the processor <NUM>.

In this embodiment of this application, a specific connection medium between the processor <NUM> and the memory <NUM> is not limited. In this embodiment of this application, the memory <NUM>, the processor <NUM>, and the display <NUM> are connected by using a bus <NUM> in <FIG>. The bus is indicated by using a bold line in <FIG>. A manner of connection between other components is merely an example for description, and imposes no limitation. The bus may be classified into an address bus, a data bus, a control bus, and the like. For ease of representation, only one bold line is used to represent the bus in <FIG>, but this does not mean that there is only one bus or only one type of bus.

The memory <NUM> may be a volatile memory (volatile memory), such as a random-access memory (random-access memory, RAM). Alternatively, the memory <NUM> may be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (hard disk drive, HDD), or a solid-state drive (solid-state drive, SSD). Alternatively, the memory <NUM> is, but not limited to, any other medium that can be used to carry or store expected program code in a form of an instruction or a data structure and can be accessed by a computer. The memory <NUM> may be a combination of the foregoing memories.

The processor <NUM> performs the program instruction in the memory <NUM>, to implement the image processing method shown in <FIG>. The method includes: sending, by the first application program, the instruction to the image processing module in the operating system, to call the image processing module to perform an image optimization algorithm; and performing, by the image processing module, image optimization processing on the to-be-displayed image, and returning an optimized image to the first application program for display.

The processor <NUM> uses the image optimization processing method in the following manner: when determining that a resolution of the to-be-displayed image is less than a first threshold, performing super-resolution processing on the to-be-displayed image.

The instruction of the first application program further includes an identifier of the first application program. The processor <NUM> is further configured to determine, based on the identifier of the first application program, that the first application program has a super-resolution processing permission.

An embodiment of this application further provides a computer readable storage medium, configured to store a computer software instruction that needs to be executed by the foregoing processor. The computer readable storage medium includes a program that needs to be executed by the foregoing processor.

An embodiment of this application further provides a computer program product. When the computer program product is executed by a computer, the computer is enabled to perform the foregoing image processing method.

In conclusion, in the embodiments of this application, the image processing module of the operating system of the image processing device is mainly improved, and an image optimization function is added, to perform image optimization processing on to-be-displayed images in different application programs at an application layer. That is, when a multimedia file in the application program calls an interface of the image processing module of the operating system for image display, an image optimization processing process of the image processing module is first performed, and an optimized image is eventually displayed. In this case, a resolution is increased, and higher definition is also achieved. In the embodiments of this application, the image processing module in the operating system is improved and it is unnecessary to develop each application program separately. A user operating an application program at the application layer is unaware of the entire image optimization process. Therefore, the method is highly reusable and more automated.

A person skilled in the art may clearly understand that, descriptions of the embodiments provided in the present invention may be reference for each other. For ease and brevity of description, for functions of the apparatuses and devices and performed steps that are provided in the embodiments of this application, refer to related descriptions in the method embodiment of the present invention.

A person skilled in the art may further understand that various illustrative logical blocks (illustrative logical block) and steps (step) that are listed in the embodiments of this application may be implemented by using electronic hardware, computer software, or a combination thereof. In order to clearly display the interchangeability (interchangeability) between the hardware and the software, functions of the foregoing various illustrative components (illustrative components) and steps have been generally described. Whether the functions are implemented by using hardware or software depends on particular applications and a design requirement of the entire system. A person of ordinary skill in the art may use various methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of the embodiments of this application.

The various illustrative logical blocks, modules, and circuits described in the embodiments of this application may implement or operate the described functions by using a general processing unit, a digital signal processing unit, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or another programmable logical apparatus, a discrete gate or transistor logic, a discrete hardware component, or a design of any combination thereof. The general processing unit may be a microprocessing unit. Optionally, the general processing unit may be any conventional processing unit, controller, microcontroller, or state machine. The processing unit may alternatively be implemented by a combination of computing apparatuses, such as a digital signal processing unit and a microprocessing unit, a plurality of microprocessing units, one or more microprocessing units with a digital signal processing unit core, or any other similar configuration.

Steps of the methods or algorithms described in the embodiments of this application may be directly embedded into hardware, a software module executed by a processing unit, or a combination thereof. The software module may be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable magnetic disk, a CD-ROM, or a storage medium of any other form in the art. For example, the storage medium may connect to a processing unit so that the processing unit can read information from the storage medium and write information to the storage medium. Optionally, the storage medium may further be integrated into a processing unit. The processing unit and the storage medium may be arranged in an ASIC, and the ASIC may be arranged in a user terminal. Optionally, the processing unit and the storage medium may be arranged in different components of the user terminal.

In one or more designs shown in examples, the functions described in the embodiments of this application may be implemented by using hardware, software, firmware, or any combination thereof. If the functions are implemented by software, these functions may be stored in a computer readable medium or be transmitted to the computer readable medium in a form of one or more instructions or code. The computer readable medium is either a computer storage medium or a communications medium that enables a computer program to move from one place to another. The storage medium may be an available medium that can be accessed by any general or special computer. For example, such a computer readable medium may include but is not limited to a RAM, a ROM, an EEPROM, a CD-ROM, or another optical disc storage, a magnetic disk storage or another magnetic storage apparatus, or any other medium that may be used to carry or store program code, where the program code is in a form of an instruction or a data structure or in a form that can be read by a general or special computer or a general or special processing unit. In addition, any connection may be appropriately defined as a computer readable medium. For example, if software is transmitted from a website, a server, or another remote resource by using a coaxial cable, an optical fiber computer, a twisted pair, a digital subscriber line (DSL) or in a wireless manner, such as infrared, radio, or microwave, the software is included in a defined computer-readable medium. The disk (disk) and the disc (disc) include a compressed disk, a laser disk, an optical disc, a DVD, a floppy disk, and a Blu-ray disc. The disk generally copies data by a magnetic means, and the disc generally copies data optically by a laser means. The foregoing combination may also be included in the computer-readable medium.

Claim 1:
An image processing method, wherein the method is applicable to an image processing device (<NUM>) having an operating system, and the method comprises:
receiving (201a), by an image processing module (<NUM>) of the operating system, an instruction of a first application program (<NUM>) to call the image processing module (<NUM>) of the operating system, wherein the instruction carries a to-be-displayed image; and
performing (202a), by the image processing module (<NUM>), image optimization processing on the to-be-displayed image and displaying an image obtained after the image optimization processing;
wherein before the performing (202a), by the image processing module (<NUM>), image optimization processing on the to-be-displayed image, the method further comprises:
determining, by the image processing module (<NUM>) that the first application program (<NUM>) has a super-resolution processing permission; and
wherein the determining, by the image display module that the first application program (<NUM>) has an image optimization processing permission comprises:
determining, by the image processing module (<NUM>), whether an identifier of the first application program (<NUM>) exists in a preset whitelist; and
if the identifier of the first application program (<NUM>) exists in the preset whitelist, determining, by the image processing module (<NUM>), that the first application program (<NUM>) has the image optimization processing permission and that super-resolution processing is permitted to be further triggered, and if the identifier of the application program is not in the whitelist, not performing, by the image processing module, super-resolution processing;
the method further comprising:
when it is determined that the first application program has the image processing permission, determining whether a resolution of the to-be-displayed image is less than a first threshold; and
when it is determined that the resolution of the to-be-displayed image is less than the first threshold, performing, by the image processing module (<NUM>), super-resolution processing on the to-be-displayed image, and when it is determined that the resolution of the to-be-displayed image is not less than the first threshold, not performing, by the image processing module, super-resolution processing.