# Web Page Content Block Identification with Extended Block Properties
## 3 Block Identification Method for Block Range and Variation Estimation
### General Idea of Web Page Division to Content Blocks with Extended Properties  
Combining the above two features (block boundaries and variations) we can achieve additional flexibility in the identified content block data in both manual and automated  
Figure 1: Example of block (menu) with three hierarchically related inner blocks (menu items).  
Figure 2: HTML code example, illustrating the range between the maximum and minimum boundaries of the blocks.  
labeling processes. This additional flexibility comes from the fact that by using these features we can reduce the total number of blocks per page by using relationship connections between them. That way when a user is manually marking blocks he or she only needs to label one variant of similar blocks, while the others will be picked up automatically. This would require less work than labeling the whole page.  
Figure 3 shows an example of how block relationships make a difference. If we had no block relationships then this code snippet alone would have 8 distinct blocks that would need to be identified to fully process this code snippet (numbered in red circles). To extract only the content-filled blocks, without taking the boundaries into account, we would need to label fewer blocks; four would be sufficient--one for the menu (red dot no. 1) and three for its items (red dot no. 4, 6 and 8). But this can lead to results that are more difficult to verify even if they are correct. The difficulty arises from the fact that during validation only exactly labeled blocks would be deemed correct, so that approach requires a lot of precision. Through the use of block relationships, we can greatly reduce the number of distinct blocks. However, we are also able to maintain information about the complete structure that allows the extraction of all data. With relationships, we would technically have only 2 blocks (marked in blue circles--one for menu and one for menu item) while all other related blocks would be accessible either by hierarchical relation or by structure variations.  
To implement web page division into blocks with extended properties, we analyze the DOM tree from top to bottom, starting with the <body> tag (see Algorithm 1). This tag would represent the most general block--web page content. The estimation of block boundaries will provide additional value, as the minimal boundary will define where the actual content and inner blocks start. At the same time, the relationship between child blocks will be able to estimate the repeating structures of variant blocks.  
The web page division into blocks solution was implemented to match the dataset data structure provided in our earlier research [22]. It takes an URL address as input and stores all the identified blocks, and the relations between them, in the database. The methods responsible for block maximum and minimum boundary identification and variation estimation are presented in further sections.  
```
input: DOM tree of the rendered web page HTML code set analyzed block to <body> set analyzed parentBlock to <body> if analyzed block is not empty then call boundaryEstimation with block and parentBlock return minBlock call getChildren with minBlock return children call getVariations with children for each children call segmentation with children end
```  
**Algorithm 1**segmentation  
Figure 3: HTML code example, illustrating the reduction of labeled blocks when instead of 8 blocks (marked with red dots), a person needs to assign labels to two (marked with blue dots) of them.