So, the decision was taken that on IP addresses with more than 100 sites, we would sample. Keeping the number of requests to be under 1000 even in the most pathological case should ensure that the survey is not banned from sites, and that the results remain respectable and accurate. For the great majority of IP addresses with fewer than 100 sites, the survey is effectively a census, taking the front page from each site. The comparison is made based on the HTML tag structure of the page, rather than the text of the whole page, as the typical scenario is that a domain registration or hosting company will use the same template tag structure for each customer, but vary some text on each site to reflect the domain name, the customer name, or the date registered, etc, which makes a strict comparison unsafe. An MD5 checksum is taken of the tag structure of each page, and this is used as the basis of the comparison, with only distinct MD5 signatures being counted.
Some companies like to use Framesets for presentation, even on inactive sites.
So where the front page contains a tag, we follow the references to take the frame pages, and compare the tag structure of those in the same way as the initial front pages. Following these links is necessary otherwise there would be a risk that all front pages consisting of references would be incorrectly assessed as inactive. The reason for doing this is that most inactive sites are automatically generated from a template, and whilst the page contents may differ between sites, the page structure remains constant. The first deployment of this methodology was in June 2000. Nearly 10 million of these sites are removed from the survey by applying the active sites methodology which found 7.5 million active sites on 3.4 million IP addresses.