Patent Description:
When faced with a scene, humans automatically process areas they are interested in and selectively ignore areas they are uninterested in. These areas that the humans are interested in are referred to as salient areas. Image saliency is an important visual feature of an image and can be used to measure importance of image content. The image saliency is also a part of image segmentation. In computer vision, the image segmentation is a process of segmenting a digital image into a plurality of segments (pixel groups, also referred to as super pixels).

Plane typesetting often involves an image and a word. A best typesetting location of the word varies with content in the image. When the image is continuously updated, the best typesetting location of the word continuously changes accordingly. Currently, when the word is typeset, some simple location determining operations can be performed based on saliency of the image to avoid a conflict between the image and the word. However, this typesetting manner is not necessarily the best in terms of a visual effect.

The document <NPL> shows an automatic label placement method for augmented reality applications.

The document <CIT> shows a system and method for hierarchical clustering for view management augmented reality.

The document <CIT> shows a system and method for image type setting and color matching.

The document <CIT> shows a system and method for image-driven view management for annotations.

The document <NPL> shows automatically placing labels in augmented reality applications.

This application may be applied to image and text typesetting of a magazine lock screen, or may be applied to image and text typesetting of a poster advertisement and the like that have a relatively high visual effect requirement, or may be applied to image and text typesetting of a sharing card such as a book list, a playlist, or an album contact. More applicable scenarios are not limited herein.

In the specification, claims, and accompanying drawings of this application, the terms "first", "second", "third", "fourth", and the like (if existent) are intended to distinguish between similar objects but do not necessarily indicate a specific order or sequence. It should be understood that the data termed in such a way are interchangeable in proper circumstances so that the embodiments of this application described herein can be implemented in orders except the order illustrated or described herein. Moreover, the terms "include", "have" and any other variants mean to cover the non-exclusive inclusion, for example, a process, method, system, product, or device that includes a list of steps or units is not necessarily limited to those expressly listed steps and units, but may include other steps or units not expressly listed or inherent to such a process, method, system, product, or device.

In a scenario in which image content such as a lock screen of a terminal changes continuously, if a single image and text typesetting manner is set on the terminal, when the image content changes, it is definitely inappropriate if a location of a word is fixed. Currently, there is a solution for performing image and text typesetting based on saliency (saliency) of an image, so that the word may be typeset in an area with relatively low saliency in the image, to avoid covering the image content by the word. However, in this typesetting manner, a visual effect is not very good, and user experience is poor. Based on this, this application discloses an image and text typesetting method, and a problem of balance between the image content and the word is considered. Referring to <FIG>, a specific implementation is as follows.

<NUM>: Determine an importance measurement value of each pixel in an image.

An image has a plurality of pixels. After obtaining the image, a terminal determines an importance measurement value of each pixel in the image. The importance measurement value is used to measure importance of the pixel. The importance measurement value may be specifically a saliency value. Generally, a higher saliency value of a pixel indicates that the pixel is more salient and is more easily observed by a user. The saliency value may be a coherent saliency value or a saliency segmentation result. For example, a grayscale value of the pixel in the image is set to <NUM> or <NUM>. A pixel whose grayscale value is larger has a larger saliency value and is more easily observed. The importance measurement value may alternatively be obtained by weighting the saliency value and any one or more of a facial recognition result and a final result. The final result is obtained after an object recognition result is sorted based on a specific object label or a semantic label. For example, weighting calculation is performed on a saliency map (saliency map) of the image and semantic information, facial information, and the like of the image to obtain the importance measurement value of each pixel.

<FIG> is a diagram of a saliency result of an image according to this application. It can be learned that some pixels of a jellyfish in the image have relatively large grayscale values, are relatively easily observed, and therefore have relatively large saliency values.

<NUM>: Obtain at least one text box area formed by at least one text box stacked on the image.

The text box area in this application refers to a text box stacked on an image interface. There may be a plurality of text boxes. Each text box has one text box area, and the text box area may be considered as an area range corresponding to a partial image obtained after the image is captured by using coordinate axes of the text box as four boundaries. It can be learned that the area range is smaller than an area range corresponding to the entire image. The obtained partial image is referred to as a background image corresponding to the text box area in this application. The background image corresponding to the text box area includes M pixels, and M is a positive integer greater than <NUM>. Because the text boxes are stacked, background images corresponding to text box areas may partially overlap.

<NUM>: Obtain, based on an importance measurement value of a pixel in a background image corresponding to each text box area, an importance measurement value of the background image corresponding to each text box area.

The importance measurement value of the background image corresponding to each text box area may be obtained through calculation based on the importance measurement value of the pixel in the background image. Specifically, there are the following several possible cases.

The importance measurement value of the background image corresponding to the text box area may be equal to a sum of the importance measurement values of the pixels in the background image, or the importance measurement value of the background image corresponding to the text box area is equal to a sum of the importance measurement values of the pixels in the background image/a quantity of the pixels in the background image, or an importance measurement value of the text box area is equal to a maximum value in the importance measurement values of all the pixels in the background image, or an importance measurement value of the text box area is equal to a mode in the importance measurement values of the pixels in the background image. More possible cases are not listed herein one by one.

A text box area with an importance measurement value as small as possible is selected. A smaller importance measurement value indicates a smaller saliency value, and the smaller saliency value indicates lower saliency and that image content is more difficult to be covered. For example, a saliency value of the jellyfish in <FIG> is relatively large, a text box area with a small saliency value is selected, to avoid covering the jellyfish by a word.

<NUM>: Obtain an importance measurement value gravity center or an importance measurement value mass center of the image based on the importance measurement value of the pixel in the image.

The importance measurement value gravity center or the importance measurement value mass center of the entire image is determined. The importance measurement value gravity center represents a gravity center, of the image, that is obtained through calculation based on the importance measurement value of each pixel in the entire image. The importance measurement value mass center represents a mass center, of the image, that is obtained through calculation based on the importance measurement value of each pixel in the entire image.

<NUM>: Determine, based on a preset principle and a location relationship between the importance measurement value gravity center and a central area of the image or a location relationship between the importance measurement value mass center and a central area of the image, information about a balance degree of each text box area relative to the image.

A central area range of the image is first determined, and the central area range may be a circle or ellipse whose center is a central point of the image.

In this embodiment, a balance degree of a text box area is used to indicate a balance degree of the text box area relative to the image. When the importance measurement value gravity center of the image is located in the central area range of the image, or the importance measurement value mass center is located in the central area range of the image, it indicates that the image is self-balanced. In this case, balance degree values of text boxes located on a horizontal central axis and a longitudinal central axis of the image are set to <NUM>, and a balance degree value of a text box at another location is set to <NUM>. A schematic diagram of the central axes of the image and the central area of the image is shown in <FIG>.

If the importance measurement value gravity center of the image is located outside the central area range of the image or even deviates greatly from the central area range of the image, or the importance measurement value mass center of the image is located outside the central area range of the image or even deviates greatly from the central area range of the image, to form visual balance between the image and the word, a balance degree value of the text box area needs to be obtained through calculation based on the importance measurement value gravity center of the image and a vector balancing principle or a lever balancing principle, or a balance degree value of the text box area needs to be obtained through calculation based on the importance measurement value mass center of the image and a vector balancing principle or a lever balancing principle. A specific calculation manner of obtaining the balance degree value of the text box area through calculation based on the vector balancing principle or the lever balancing principle is as follows:
The vector balancing principle is specifically as follows: A first vector is formed from coordinates of a geometric center of a text box to coordinates of a central point of an image. A second vector is formed from coordinates of an importance measurement value mass center of the image to the coordinates of the central point of the image (or a second vector is formed from coordinates of an importance measurement value gravity center of the image to the coordinates of the central point of the image). A balance degree value of the text box is determined based on values and directions of the two vectors. When the two vectors are equal in value and opposite in direction, a maximum balance degree of the text box is <NUM>, and in another case, the balance degree ranges from <NUM> to <NUM>. It may be understood that a value range <NUM> to <NUM> herein is merely an example, the value may alternatively range from <NUM> to <NUM>, and more value ranges are not limited herein.

The lever balancing principle is specifically as follows: A distance from a geometric center of a text box to a central point of an image is a first arm of force. A distance from an importance measurement value mass center of the image to the central point of the image is a second arm of force (or a distance from an importance measurement value gravity center of the image to the central point of the image is a second arm of force). Text box mass = preset text box density x a text area. Image mass is obtained through calculation by using image saliency segmentation. When the first arm of force x the text box mass = the second arm of force x the image mass, a maximum balance degree of the text box is <NUM>, and in another case, the balance degree ranges from <NUM> to <NUM>. It may be understood that a value range <NUM> to <NUM> herein is merely an example, and more value ranges are not limited herein.

Herein, the balance degree of the text box area may be determined based on another image balancing principle in addition to the vector balancing principle and the lever balancing principle. This is not limited in this application.

It is found, after the balance degree value of the text box is determined based on the lever balancing principle or the vector balancing principle, that when the importance measurement value gravity center (or the importance measurement value mass center) of the image is located at an upper left part of the image, a balance degree value of a text box area at a lower right part of the image is relatively large; when the importance measurement value gravity center (or the importance measurement value mass center) of the image is located at an upper right part of the image, a balance degree value of a text box area at a lower left part of the image is relatively large; and similarly, when the importance measurement value gravity center or the importance measurement value mass center of the image is located in another direction, a balance degree value of a text box area in a direction opposite to the importance measurement value gravity center or the importance measurement value mass center of the image is relatively large. Referring to <FIG>, the central area of the image is shown as an ellipse in the figure. The importance measurement value gravity center of the image is located outside the central area of the image, and the importance measurement value gravity center of the image is located at the upper left part. In this case, a balance degree value of a text box in a lower right area is relatively large, and a text box area is preferentially selected in this area to typeset a word.

In this embodiment, a text box area balanced with the image is preferentially selected to typeset the word, so that typesetting of the image and the word may generate visual balance, thereby improving user experience.

<NUM>: Select, from the at least one text box area based on the importance measurement value of the background image corresponding to each text box area and the information about the balance degree of each text box area, one text box area to typeset the word.

Weighting calculation is performed on a balance degree priority of the text box area and the importance measurement value of the text box area to obtain a calculation result, and then the text box area is selected based on the calculation result to typeset the word. For example:<MAT> by the terminal based on an actual situation.

It can be learned that the importance measurement value of the text box area is considered when the text box area is selected to typeset the word in this application. Therefore, a text box area with a relatively small saliency value may be selected to avoid covering the image content. In addition, a balance degree between the text box area and the image is considered, so that the visual balance can be ensured. A best visual effect of the text box area is obtained by combining the importance measurement value and the balance degree, and user experience is improved.

Further, in this application, whether a mask needs to be added to a background of the word may be further determined. Details are as follows.

A color feature of a background image corresponding to a text box area in which the word is located is determined based on an image feature, an information entropy, a saliency value, or a color distribution of the background image corresponding to the text box area in which the word is located. The image feature is specifically a color feature of the image. When the color feature of the background image corresponding to the text box area is a preset color feature of a mottled type, the mask is added to the background of the word. It should be noted that a size of an area to which the mask is added is determined by a size of the text box area. Generally, a mask area is larger than the text box area but smaller than an area in which the entire image is located. For example, as shown in <FIG>, a lower right text box is selected to typeset the word, and the mask area is slightly larger than the text box area.

Further, in this application, a dominant color of the background image corresponding to the text box area in which the word is located may further be determined, and a shadow color and a highlight color are derived from the determined dominant color in a dark-level calculation manner in HSV (Hue, Saturation, Value) or LCH (lightness, chroma, hue) space.

Lightness of the background image corresponding to the text box area in which the word is located is determined. Then, a text in the text box is set to the highlight color or the shadow color based on the lightness. If the mask is added, the mask area may further be set to the highlight color or the shadow color. Specifically, when the lightness of the background image corresponding to the text box area in which the word is located belongs to a preset lighter chroma set, whether the mask is added to the background of the word is further determined. If the mask is added to the background of the word, the text in the text box is set to the shadow color, and the mask area is set to the highlight color. If the mask is not added to the background of the word, the text in the text box area is set to the shadow color. When the lightness of the background image corresponding to the text box area in which the word is located belongs to a preset darker chroma set, whether the mask is added to the background of the word is further determined. If the mask is added to the background of the word, the text in the text box is set to the highlight color, and the mask area is set to the shadow color. If the mask is not added to the background of the word, the text in the text box area is set to the highlight color. A better visual effect can be achieved in the foregoing setting manner.

<FIG> shows lock screen pictures of the terminal. The user may browse the pictures through flicking. The terminal may also dynamically update the pictures in real time based on stored pictures. For each picture, a text box area with a best visual effect is selected based on an image and text typesetting principle of this application, to typeset a word.

The image and text typesetting method in this application is described above. Referring to <FIG>, a structure of the terminal in this application is described. A terminal <NUM> includes:
a receiver <NUM>, a transmitter <NUM>, a processor <NUM>, and a memory <NUM> (there may be one or more processors <NUM> in the terminal <NUM>, and one processor is used as an example in <FIG>), where in some embodiments of this application, the receiver <NUM>, the transmitter <NUM>, the processor <NUM>, and the memory <NUM> may be connected through a bus or in another manner, and in <FIG>, a bus connection is used as an example.

The memory <NUM> may include a read-only memory and a random access memory, and provide instructions and data to the processor <NUM>. A part of the memory <NUM> may further include a non-volatile random access memory (English full name: non-volatile random access memory, NVRAM for short). The memory <NUM> stores an operating system and operation instructions, an executable module or a data structure, a subset thereof, or an extended set thereof. The operation instructions may include various operation instructions, used to implement various operations. The operating system may include various system programs, to implement various basic services and process hardware-based tasks.

The processor <NUM> controls an operation of a network device, and the processor <NUM> may also be referred to as a central processing unit (English full name: central processing unit, CPU for short). In a specific application, components of the network device are coupled together through a bus system. In addition to a data bus, the bus system may further include a power bus, a control bus, a status signal bus, and the like. However, for clear description, various types of buses in the figure are referred to as the bus system.

The method disclosed in the foregoing embodiment of this application may be applied to the processor <NUM>, or may be implemented by the processor <NUM>. The processor <NUM> may be an integrated circuit chip and has a signal processing capability. In an implementation process, steps in the foregoing method can be implemented by using a hardware integrated logical circuit in the processor <NUM>, or by using instructions in a form of software. The processor <NUM> may be a general-purpose processor, a digital signal processor (English full name: digital signal processing, DSP for short), an application-specific integrated circuit (English full name: application specific integrated circuit, ASIC for short), a field programmable gate array (English full name: field-programmable gate array, FPGA for short) or another programmable logical component, a discrete gate or transistor logic device, or a discrete hardware component. The processor <NUM> may implement or perform the method, the steps, and logical block diagrams that are disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. Steps of the method disclosed with reference to the embodiments of this application may be directly executed and accomplished by a hardware decoding processor, or may be executed and accomplished by using a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory <NUM>, and the processor <NUM> reads information in the memory <NUM> and completes the steps in the foregoing method in combination with hardware of the processor.

The receiver <NUM> may be configured to receive input digit or character information, and generate signal input related to a related setting and function control of the network device. The transmitter <NUM> may include a display device such as a display. The transmitter <NUM> may be configured to output digit or character information through an external interface.

In this embodiment of this application, the processor <NUM> is configured to perform the foregoing method.

Referring to <FIG>, another structure of the terminal in this application is described. A terminal <NUM> includes:.

Optionally, the importance measurement value of the background image corresponding to the text box area = a sum of the importance measurement values of the pixels in the background image.

Alternatively,
the importance measurement value of the background image corresponding to the text box area = a sum of the importance measurement values of the pixels in the background image/a quantity of the pixels in the background image.

Alternatively,
the importance measurement value of the background image corresponding to the text box area = a maximum value or a mode in the importance measurement values of the pixels in the background image.

Optionally, the processing unit <NUM> is specifically configured to: when the importance measurement value gravity center is located in a central area range of the image, or the importance measurement value mass center is located in a central area range of the image, set a balance degree value of a text box area located on a central axis of the image to <NUM>, and set a balance degree value of a text box area that is not located on the central axis of the image to <NUM>, where the text box area located on the central axis of the image and the text box area that is not located on the central axis of the image belong to the at least one text box area; and
when the importance measurement value gravity center is not located in the central area range of the image, or the importance measurement value mass center is not located in the central area range of the image, obtain the balance degree value of each text box area based on a vector balancing principle or a lever balancing principle.

Optionally, the text box selecting unit <NUM> is specifically configured to:.

Optionally, the importance measurement value is obtained by weighting a saliency value and any one or more of a facial recognition result, an object recognition result, and semantic information. Alternatively, the importance measurement value is obtained by weighting a saliency segmentation result and any one or more of a facial recognition result, an object recognition result, and semantic information. Alternatively, the importance measurement value is specifically a saliency value.

Optionally, the terminal further includes:.

Optionally, the processing unit <NUM> is further configured to: determine a dominant color of the target background image;.

It should be noted that content such as information exchange between the modules/units of the apparatus and the execution processes thereof is based on the same idea as the method embodiments of this application, and produces the same technical effects as the method embodiments of this application. For the specific content, refer to the foregoing description in the method embodiments of this application, and details are not described herein again.

In addition, it should be noted that the apparatus embodiments are merely examples. The modules described as separate parts may or may not be physically separate, and modules displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all the modules may be selected according to an actual need to achieve the objectives of the solutions of the embodiments. In addition, in the accompanying drawings of the apparatus embodiments provided in this application, connection relationships between modules indicate that the modules have communication connections with each other, which may be specifically implemented as one or more communications buses or signal cables.

Based on the description of the foregoing implementations, a person skilled in the art may clearly understand that this application may be implemented by software in addition to necessary universal hardware, or by dedicated hardware, including a dedicated integrated circuit, a dedicated CPU, a dedicated memory, a dedicated component, and the like. Generally, any functions that can be performed by a computer program can be easily implemented by using corresponding hardware. Moreover, a specific hardware structure used to achieve a same function may be of various forms, for example, in a form of an analog circuit, a digital circuit, a dedicated circuit, or the like. Based on such an understanding, the technical solutions of this application essentially or the part contributing to the conventional technology may be implemented in a form of a software product. The software product is stored in a readable storage medium, such as a floppy disk, a USB flash drive, a removable hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or a compact disc of a computer, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform the method described in the embodiments of this application.

Claim 1:
An image and text typesetting computer-implemented method, comprising:
determining (<NUM>) an importance measurement value of each pixel in an image;
obtaining (<NUM>) at least one text box area formed by at least one text box overlapped on the image, wherein the text box area is an area range of a background image obtained after the image is captured by using a coordinate axis of the text box as a boundary, and a background image corresponding to each of the at least one text box area comprises at least one pixel;
obtaining (<NUM>), based on an importance measurement value of the pixels in the background image corresponding to each text box area, an importance measurement value of the background image corresponding to each text box area;
obtaining (<NUM>) an importance measurement value gravity center or an importance measurement value mass center of the image based on the importance measurement values of the pixels in the image;
determining (<NUM>), based on a preset principle and a location relationship between the importance measurement value gravity center and a central area of the image or a location relationship between the importance measurement value mass center and a central area of the image, information about a balance degree of each text box area relative to the image; and
selecting (<NUM>), from the at least one text box area based on the importance measurement value of the background image corresponding to each text box area and the information about the balance degree of each text box area, one text box area to typeset a word.