# Pattern-Based and Visual Analytics for Visitor Analysis on Websites
## 5 Traffic Segmentation and Characterization
### Case Study Scenario: Humans vs. Bots  
Often, the first step of visitor analysis on web sites is to perform a data cleaning process to eliminate bot visits from the analysis. We present the bot versus human analysis, as an example of how pattern mining can be applied to characterize two visitor segments. We used data mining techniques to extract patterns which marketing teams can observe to get interesting trends from both types of visitors. We used the dataset described in Section 3. For this scenario, the pattern mining algorithm will work with two classes: human and bot; which were obtained using a one-class classifier (BaggingRandomMiner [44]). Next, we will show the extracted patterns separating human from bot traffic from a specific day of interest. Note that the analysis does not have to cover only a particular day, it can cover the desired period of interest. For example, if a marketing campaign lasted four days, the patterns can be extracted from that time span. Alternatively, it may be of interest to feed the system with visits from certain hours; for example, as will be seen shortly, patterns may outline an interesting subset of visits, which is of interest to analysts. Patterns with high support for one class are of interest because they show a general behavior of that class. Also, patterns without necessarily high support but with zero support for the other class are also interesting because they show subgroups of visitors. First, we have a pair of related patterns that characterize the normal class:  
1. \(\text{country}=\text{``}\textit{Mexico''}\wedge\text{hour}\leq 10\wedge\text{ agentOS}\
eq\text{``}\textit{Other''}\)
2. \(\text{country}=\text{``}\textit{Mexico''}\wedge\text{hour}>10\)  
Both patterns have a support of zero for the anomalous class, which means that they do not cover bot visitors. Additionally, pattern A has a support of 0.1107 and pattern B of 0.8574. Given the nature of these particular patterns (the intersection of the visitors covered by both patterns is zero), together, they account for 96.81% of the human visitors. We can conclude that for this day, at least 96% of the human visitors come from Mexico. Many more visits (85.74%) are registered after 10 a.m. than those registered before this time (11.07%). This can serve as an indicator that only a few users visit the page early in the morning, most users visit after 10 a.m. Another pattern with zero support for the bot class and high support for the human class is:  
1. \(\text{country}=\text{``}\textit{Mexico''}\wedge\text{referer}\
eq\text{``}\text{? ''}\)  
Pattern C has a support of 0.9618 for the normal class. This pattern states a high percentage of the users (96.18%) come from Mexico, as we already know by pattern A and B; but it is interesting because it adds the information that these visitors arrive through a referrer. We found that our one-class classifier was able to find visits from two known crawlers, even when the provided log files were pre-selected as human behavior. Next, we show some patterns with zero support for the human class, which means that only bots follow each of them:  
1. \(\text{agentBrowser}=\text{``}\textit{Sogouwebspider''}\)* country \(\
eq\)_\"Mexico\"_\(\wedge\) agentOS \(\
eq\)_\"Other_\" \(\wedge\) agentBrowser = _\"BingPreview\"_
* city \(\
eq\)_\"Redmond\"_
* city \(\
eq\)_\"Redmond\"_\(\wedge\) subdivision = _\"Beijing\" \(\wedge\)url \(\
eq\)_\"robots.txt\"_
* city \(=\)_\"Sunnyvale\"_\(\wedge\) subdivision \(\
eq\)_\"Beijing\"_\(\wedge\)referer \(\
eq\)_\"?\"_  
In the analyzed day, Sogou web spider (Chinese search engine crawler) and Bing Preview (used to generate page snapshots) appeared in patterns D and E, with a support of 0.2372, and 0.2146, respectively. Other patterns describing bot visitors, like F, G, and H, were inclined to geolocation features, indicating us a fraction of the bots come from Beijing (pattern G support: 0.2514), Redmond (pattern F support: 0.2571), and Sunnyvale (pattern H support: 0.1356).