As discussed for the large site analysis, obtaining this information is not straightforward and involves considerable time evaluating each site. We randomized our listing of 17,000 search site candidates. We then proceeded to work through this list until 100 sites were fully characterized. Total record counts as reported by the site itself. This involved inspecting relating pages on the site including help sections, site FAQs, etc.
Record counts as provided by the site's own search function.
Some site searches provide total record counts for all queries submitted. Exactly 700 sites were inspected in their randomized order to obtain the 100 fully characterized sites. Random queries were issued to the searchable database with results reported out as HTML pages. A minimum of ten of these were generated, saved to disk, and then averaged to determine the mean site page size. In a few cases, such as bibliographic databases, multiple records were reported on a single HTML page.