Except for the Web sites on the Intranets, many of the nonpublic pages are accessed through a gateway page. These gateway pages are usually public pages and will be included in the sample. The gateway pages will provide sufficient information to estimate the amount of nonpublic material and to identify the common types of restricted pages. The World Wide Web is an important and rapidly growing information resource. However, relatively little is known about the collection characteristics of the Web pages.
Reliable statistics are rare.
The sampling procedure described in this paper is based on clustering sampling methodology and can be used to collect a representative sample of publicly accessible Web pages. The resulting analysis of the sample is expected to yield comprehensive and accurate statistics on both the size and characteristics of Web accessible information. Cochran, Willian G. Sampling Techniques.