Abstract:
The invention discloses a system for identifying a main body of a webpage, which comprises: a webpage parse &amp; layout module configured to parse source codes of the webpage, perform a layout calculation on the parsed result, and generate a DOM tree of the webpage; a node identification module configured to traverse starting from the root node of the DOM tree, and identify a main body node and a spam word node in the DOM tree; a floor division module configured to divide the identified main body node according to floors of the webpage; and a mobile terminal page generation module configured to generate a mobile terminal page. After the invention identifies and extracts the content of a traditional internet webpage, it may effectively extract a BBS main body, a news main body and a comment, and restore a presentation feature of “divided floors” of the content of a main body in the original webpage, of which the presentation effect maintains the original “multi-floor” feature, so as to provide a user with an excellent reading experience.

Description:
FIELD OF THE INVENTION 
       [0001]    The invention relates to the field of internet, and in particular, to a system and method for identifying the floor of a main body of a webpage. 
       BACKGROUND OF THE INVENTION 
       [0002]    With the development and popularization of mobile terminals, people more and more use a mobile terminal to browse a webpage. However, since most websites on the internet do not make a special treatment on the webpage presentation of a mobile terminal, deformations of the presentation of most webpages occur on the mobile terminal, which leads to an extremely poor reading experience for a user. 
         [0003]    The current methods for improving a user&#39;s reading experience are to extract and rearrange main bodies of a webpage, and then re-present them to the user. For a news and information webpage with massive content, the effect is good, but user comments will be discarded; for a forum in which a main body is divided into multiple “floors”, etc., the effect is worse: only the main body of a certain floor can be identified, or the main body cannot be identified. Spam word information in a source webpage is not removed, and the content of the webpage does not have a fixed effect, and the effects of the generated webpage and the source webpage will appear. 
       SUMMARY OF THE INVENTION 
       [0004]    In view of the above problems, the invention is proposed to provide a system and method for identifying the floor of a main body of a webpage which overcome the above problems or at least in part solve or mitigate the above problems. 
         [0005]    According to an aspect of the invention, there is provided a system for identifying a main body of a webpage, which comprises: a webpage parse &amp; layout module configured to parse source codes of the webpage, perform a layout calculation on the parsed result, and generate a DOM tree of the webpage; a node identification module configured to traverse starting from the root node of the DOM tree, and identify a main body node and/or a spam word node in the DOM tree; and a floor division module configured to divide the identified main body node according to floors of the webpage. 
         [0006]    According to another aspect of the invention, there is provided a method for identifying a main body of a webpage, which comprises: parsing source codes of the webpage, performing a layout calculation on the parsed result, and generating a DOM tree of the webpage; traversing starting from the root node of the DOM tree, and identifying a main body node and/or a spam word node in the DOM tree; and dividing the identified main body node according to floors of the webpage. 
         [0007]    According to yet another aspect of the invention, there is provided a computer program comprising a computer readable code which causes a server to perform the method for identifying a main body of a webpage according to any of claims  15 - 28 , when said computer readable code is running on the server. 
         [0008]    According to still another aspect of the invention, there is provided a computer readable medium storing the computer program as claimed in claim  29  therein. 
         [0009]    The beneficial effects of the invention lie in that: 
         [0010]    After the invention identifies and extracts the content of a traditional internet webpage, it may effectively extract a BBS main body, a news main body and comments, and restore a presentation feature of “divided floors” of the content of a main body in the original webpage, of which the presentation effect maintains the original “multi-floor” feature, so as to provide a user with an excellent reading experience. 
         [0011]    The above description is merely an overview of the technical solutions of the invention. In the following particular embodiments of the invention will be illustrated in order that the technical means of the invention can be more clearly understood and thus may be embodied according to the content of the specification, and that the foregoing and other objects, features and advantages of the invention can be more apparent. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0012]    Various other advantages and benefits will become apparent to those of ordinary skills in the art by reading the following detailed description of the preferred embodiments. The drawings are only for the purpose of showing the preferred embodiments, and are not considered to be limiting to the invention. And throughout the drawings, like reference signs are used to denote like components. In the drawings: 
           [0013]      FIG. 1  shows schematically a structure diagram of a system according to an embodiment of the invention; 
           [0014]      FIG. 2  shows schematically a flow chart of a method according to an embodiment of the invention; 
           [0015]      FIG. 3  shows schematically a DOM tree generated according to an embodiment of the invention; 
           [0016]      FIG. 4  shows schematically a diagram of a mobile terminal webpage generated according to the DOM tree of  FIG. 3 ; 
           [0017]      FIG. 5  shows schematically a block diagram of a server for performing a method according to the invention; and 
           [0018]      FIG. 6  shows schematically a storage unit for retaining or carrying a program code implementing a method according to the invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0019]    In the following the invention will be further described in connection with the drawings and the particular embodiments. 
         [0020]    A structure diagram of a system according to an embodiment of the invention is as shown in  FIG. 1 . 
         [0021]    The webpage parse &amp; layout module  100  parses and performs a layout calculation on source codes of a webpage. When parsing HTML source codes and laying out, an HTML parse engine is adopted, and a commonly used open source HTML parse engine is e.g., webkit. The parse &amp; layout is based on a label in the source codes of the webpage, which may be based on, but not limited to, the div label, to generate a DOM tree of the webpage, and calculate the position and the height shown by individual nodes when the webpage is presented. One generated DOM tree is as shown in  FIG. 3 . 
         [0022]    Since on a mobile terminal, the dynamic effect of an internet webpage is difficult to be displayed, the dynamic effect needs to be given up in the process of generating the DOM tree, and only a link to pictures and the text format of a main body are kept. 
         [0023]    The node identification module  200  traverses the whole DOM tree starting from the body node, and identifies the main body content and the spam word content by the algorithm which can classify data rules, such as a typical decision tree algorithm. 
         [0024]    The node identification module  200  comprises a statistics module, a comparison module and a main body identification module. First, the statistics module calculates the node distribution value, the text density and the spam word density of the page of each webpage; then, the comparison module compares the node distribution value, the text density and the spam word density with a corresponding preset threshold; and finally, the main body identification module identifies the content in the DOM tree, of which the node distribution value, the text density and the spam word density fall within the threshold, as a main body. Therein, the node distribution represents the composition of child nodes of a node, for example, the number of individual labels, such as div, img, table, etc., the proportion of the labels in the child nodes; the text density represents an average text length obtained by dividing the text length in a node by the number of its child nodes; and the spam word (non-body vocabulary) density represents a value of the division of the length of all the ad words in a node by the length of all the texts in the node. The spam word is identified based on a dictionary, and maintained manually, for example, a word and a phrase such as print preview, support, hot comments, no hot comments yet, etc., which are irrelevant to a main body in the webpage. 
         [0025]    From the above three features, a threshold is obtained according to the decision tree algorithm, and nodes within the range of the threshold are all identified as a main body, and others are identified as spam words. 
         [0026]    The floor division module comprises a position division module and a feature word division module. 
         [0027]    The position division module performs a floor division and identification according to the path and positional relationship of a main body node on the DOM tree, and the rules which are based on when dividing are as follows. 
         [0028]    1. if two main body nodes are adjacent to each other on the DOM tree, then the two nodes belong to one and the same floor. 
         [0029]    As shown in  FIG. 3 , br represents a line break, and the br label is an empty label. The main body node  1  and the main body node  2  have a common father node div 1 , and the main body node  1  and the main body node  2  are adjacent to each other, and therefore the main body node  1  and the main body node  2  may be identified as nodes in one and the same floor. 
         [0030]    2. if one main body node and other main body nodes which have already been determined as belonging to one and the same floor have a common father node, then these main body nodes belong to one and the same floor. 
         [0031]    For example, the main body node  3  in  FIG. 3  and the main body node  2 , the main body node  1  have a common father node div 1 , and the main body node  2  and the main body node  1  have been determined as belonging to one and the same floor, therefore, the main body node  3  also belongs to the same floor. 
         [0032]    3. if a common father node of two main body nodes is body, then the two main body nodes are divided into different floors. 
         [0033]    For example, for the main body node  1  and the main body node  4 , their paths in the DOM tree are respectively: 
         [0034]    main body  1 →div 1 →body 
         [0035]    main body  4 →div 3 →body 
         [0036]    The common father node of their paths is body, and thereby they should be identified as being at different floors. 
         [0037]    4. if the relationship between main body nodes is not comprised in the above situations, then they are divided into different floors. 
         [0038]    The feature word division module performs a division primarily according to a feature word in a node, for example, a BBS main body or a news &amp; information review, i.e. 
         [0039]    the content published by the author, is presented simultaneously together with relevant information of the author, and they appear alternately, usually as follows: 
         [0040]    author information→main body→author information→main body→author information→main body . . . 
         [0041]    A further “floor” division is performed on a main body by identifying a key word (e.g., publish time, register time, etc.) in a non-body node indicating the information of an author. 
         [0042]    The mobile terminal page generation module comprises a layout generation module configured to re-lay out the content of a main body node according to its divided floors, and generating a mobile terminal page. In the above process, according to the DOM tree as shown in  FIG. 3 , the floor distribution result of the main body nodes is as shown in  FIG. 4 , namely, 
         [0043]    floor  1 : main body  1 , main body  2 , main body  3 ; 
         [0044]    floor  2 : main body  4 ; 
         [0045]    floor  3 : main body  5 , main body  6 . 
         [0046]    A flow chart of the method provided by the invention is as shown in  FIG. 2 . 
         [0047]    S 102 : performing a parse &amp; layout calculation on source codes of a webpage. When parsing HTML source codes and laying out, an HTML parse engine is adopted, and a commonly used open source HTML parse engine is e.g., webkit. The parse &amp; layout is based on a label in the source codes of the webpage, primarily the div label, to generate a DOM tree of the webpage, and calculate the position and the height shown by individual nodes when the webpage is presented. One generated DOM tree is as shown in  FIG. 3 . 
         [0048]    Since on a mobile terminal, the dynamic effect of an internet webpage is difficult to be displayed, the dynamic effect needs to be given up in the process of generating the DOM tree, and only a link to pictures and the text format of a main body are kept. 
         [0049]    S 104 : traversing the whole DOM tree starting from the body node, and identifying the main body content and the spam word content, by the algorithm which can classify data rules, such as a typical decision tree algorithm. 
         [0050]    First, the node distribution value, the text density and the spam word density of the page of each webpage are calculated; then, the node distribution value, the text density and the spam word density are compared with a preset threshold respectively; and finally, the content in the DOM tree, for which the threshold is not exceeded, is identified as a main body. 
         [0051]    Therein, the node distribution represents the composition of child nodes of a node, for example, the number of an individual label, such as div, img, table, etc., the proportion of the labels in the child nodes; the text density represents an average text length obtained by dividing the text length in a node by the number of its child nodes; and the spam word (non-body vocabulary) density represents a value of the division of the length of all the ad words in a node by the length of all the texts in the node. The spam word is identified based on a dictionary, and maintained manually, for example, a word and a phrase such as print preview, support, hot comments, no hot comments yet, etc., which are irrelevant to a main body in the webpage. 
         [0052]    From the above three features, a threshold is obtained according to the decision tree algorithm, and nodes within the range of the threshold are all identified as a main body, and others are identified as spam words. 
         [0053]    S 106 : dividing the identified main body node according to floors of the webpage, and the used method comprises division by position and division by feature word. Division by position is to perform a floor division and identification according to the path and positional relationship of a main body node on the DOM tree, and the rules which are based on when dividing are as follows. 
         [0054]    1. if two main body nodes are adjacent to each other on the DOM tree, then the two nodes belong to one and the same floor. 
         [0055]    As shown in  FIG. 3 , br represents a line break, and the br label is an empty label. The main body node  1  and the main body node  2  have a common father node div 1 , and the main body node  1  and the main body node  2  are adjacent to each other, and therefore the main body node  1  and the main body node  2  may be identified as nodes in one and the same floor. 
         [0056]    2. if one main body node and other main body nodes which have already been determined as belonging to one and the same floor have a common father node, then these main body nodes belong to one and the same floor. 
         [0057]    For example, the main body node  3  in  FIG. 3  and the main body node  2 , the main body node  1  have a common father node div 1 , and the main body node  2  and the main body node  1  have been determined as belonging to one and the same floor, therefore, the main body node  3  also belongs to the same floor. 
         [0058]    3. if a common father node of two main body nodes is body, then the two main body nodes are divided into different floors. 
         [0059]    For example, for the main body node  1  and the main body node  4 , their paths in the DOM tree are respectively: 
         [0060]    main body  1 →div 1 →body 
         [0061]    main body  4 →div 3 →body 
         [0062]    The common father node of their paths is body, and thereby they should be identified as being at different floors. 
         [0063]    4. if the relationship between main body nodes is not comprised in the above situations, then they are divided into different floors. 
         [0064]    Division by feature word is to perform a division according to a feature word in a main body. For example, a BBS main body, a news &amp; information review, i.e. the content published by the author, is presented simultaneously together with relevant information of the author, and they appear alternately, usually as follows: 
         [0065]    author information→main body→author information→main body→author information→main body . . . 
         [0066]    A further “floor” division is performed on a main body by identifying a key word (e.g., publish time, register time, etc.) in a non-body node indicating the information of an author. 
         [0067]    A mobile terminal page is generated, wherein the content of a main body node is re-laid out according to its divided floors, and then a mobile terminal page is generated. In the above process, according to the DOM tree as shown in  FIG. 3 , the floor distribution result of the main body nodes is as shown in  FIG. 4 , namely, 
         [0068]    floor  1 : main body  1 , main body  2 , main body  3 ; 
         [0069]    floor  2 : main body  4 ; 
         [0070]    floor  3 : main body  5 , main body  6 . 
         [0071]    It should be noted that, in the individual components of the controller of the invention, the components therein are divided logically according to the functionality to be realized by them, however, the invention is not limited thereto, and the individual components may be re-divided or combined as needed, for example, some components may be combined into a single component, or some components may be further decomposed into more sub-components. 
         [0072]    Embodiments of the individual components of the invention may be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that, in practice, some or all of the functions of some or all of the components in the system for identifying a main body of a webpage according to individual embodiments of the invention may be realized using a microprocessor or a digital signal processor (DSP). The invention may also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for carrying out a part or all of the method as described herein. Such a program implementing the invention may be stored on a computer readable medium, or may be in the form of one or more signals. Such a signal may be obtained by downloading it from an Internet website, or provided on a carrier signal, or provided in any other form. 
         [0073]    For example,  FIG. 5  shows a server which may carry out the method for identifying a main body of a webpage according to the invention, e.g., an application server. The server traditionally comprises a processor  510  and a computer program product or a computer readable medium in the form of a memory  520 . The memory  520  may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory  520  has a memory space  530  for a program code  531  for carrying out any method steps in the methods as described above. For example, the memory space  530  for a program code may comprise individual program codes  531  for carrying out individual steps in the above methods, respectively. The program codes may be read out from or written to one or more computer program products. These computer program products comprise such a program code carrier as a hard disk, a compact disk (CD), a memory card or a floppy disk. Such a computer program product is generally a portable or stationary storage unit as described with reference to  FIG. 6 . The storage unit may have a memory segment, a memory space, etc. arranged similarly to the memory  520  in the server of  FIG. 5 . The program code may for example be compressed in an appropriate form. In general, the storage unit comprises a computer readable code  531 ′, i.e., a code which may be read by e.g., a processor such as  510 , and when run by a server, the codes cause the server to carry out individual steps in the methods described above. 
         [0074]    “An embodiment”, “the embodiment” or “one or more embodiments” mentioned herein implies that a particular feature, structure or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. In addition, it is to be noted that, examples of a phrase “in an embodiment” herein do not necessarily all refer to one and the same embodiment. 
         [0075]    In the specification provided herein, a plenty of particular details are described. However, it can be appreciated that an embodiment of the invention may be practiced without these particular details. In some embodiments, well known methods, structures and technologies are not illustrated in detail so as not to obscure the understanding of the specification. 
         [0076]    It is to be noted that the above embodiments illustrate rather than limit the invention, and those skilled in the art may design alternative embodiments without departing the scope of the appended claims. In the claims, any reference sign placed between the parentheses shall not be construed as limiting to a claim. The word “comprise” does not exclude the presence of an element or a step not listed in a claim. The word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of a hardware comprising several distinct elements and by means of a suitably programmed computer. In a unit claim enumerating several devices, several of the devices may be embodied by one and the same hardware item. Use of the words first, second, and third, etc. does not mean any ordering. Such words may be construed as naming. 
         [0077]    Furthermore, it is also to be noted that the language used in the description is selected mainly for the purpose of readability and teaching, but not selected for explaining or defining the subject matter of the invention. Therefore, for those of ordinary skills in the art, many modifications and variations are apparent without departing the scope and spirit of the appended claims. For the scope of the invention, the disclosure of the invention is illustrative, but not limiting, and the scope of the invention is defined by the appended claims.