Metadata Statistics for a Large Web Corpus
Peter Mika

Tim Potter

Yahoo! Research
Diagonal 177
Barcelona, Spain

Yahoo! Research
Diagonal 177
Barcelona, Spainpmika@yahoo-inc.com

tep@yahoo-inc.com

ABSTRACT
We provide an analysis of the adoption of metadata standards on the Web based a large crawl of the Web. In particular, we look at what forms of syntax and vocabularies
publishers are using to mark up data inside HTML pages.
We also describe the process that we have followed and the
difficulties involved in web data extraction.

1.

INTRODUCTION

Embedding metadata inside HTML pages is one of theways to publish structured data on the Web, often preferred by publishers and consumers over other methods of
exposing structured data, such as publishing data feeds,
SPARQL endpoints or RDF/XML documents. Publishers
prefer this method due to the ease of implementation andmaintenance: since most webpages are dynamically generated, adding markup simply requires extending the templatethat produces the pages. Consumers such as search engines
are already accustomed to processing HTML and extraction
fits naturally in their processing pipelines. The close coupling of the raw data and the HTML presentation of thedata has other advantages, among others it makes sure thatthe the raw data and the end-user presentation show thesame.
In this paper, we describe the method by which we extracted metadata from a large web corpus and present somestatistics. Results from similar experiments have been already published, so we also discuss the difficulty in comparing numbers across the various studies.

2.

RELATED WORK

Previous studies have reported results on the usage of embedded metadata, including Bizer et al. at http://www.
webdatacommons.org/. We also published an earlier analysis on a different corpus collected by Yahoo! Search 1 . There1
http://tripletalk.wordpress.com/2011/01/25/
rdfa-deployment-across-the-web/

Copyright is held by the author/owner(s).
LDOW2012, April 16, 2012, Lyon, France.

are a number of factors that complicate the comparison of
results. First, different studies use different web corpora.
Our earlier study used a corpus collected by Yahoo!’s web
crawler, while the current study uses a dataset collected by
the Bing crawler. Bizer et al. analyze the data collected by
http://www.commoncrawl.org, which has the obvious advantage that it is publicly available. Second, the extraction
methods may differ. For example, there are a multitude of
microformats (one for each object type) and although mostsearch engines and extraction libraries support the popular
ones, different processors may recognize a different subset.
Unlike the specifications of microdata and RDFa publishedby the RDFa, the microformat specifications are also rather
informal and thus different processors may extract different
information from the same page. Further, even if the same
information is extracted, the conversion of this information
to RDF may differ across implementations. Third, different
extractors may be lenient in accepting particular mistakes in
the markup, leading to more or less information extracted.

3.

ANALYSIS

We take as our starting point a sufficiently large sampleof the web crawl produced by Bing’s web crawler during
January, 2012. After retaining information resources witha content type that includes text/html, we get a data setof 3,230,928,609 records with only the three fields required
for analysis, the URL of the page, the content type and the
downloaded content. In case the crawler arrived to a pageby following a (chain of) redirects, we considered the targetof the redirect as the URL.
We perform our analysis in two steps. First, we use regular expression patterns to detect metadata in web pages.
We use the same patterns proposed by Bizer et al., but westrengthen the pattern for detecting RDFa. In the form proposed by the authors it allows any page that contains about
followed by whitespace and an equal sign;we limit this pattern to require that the equal sign be followed by whitespace
and a single or double quote. We also introduce a new pattern to specifically detect webpages using the Open Graph
Protocol Second, identified by the word property followed by
optional whitespace, single or double quote, optional whitespace and og:. For this analysis, we filter out pages larger
than 3MB and where the character set can not be identified. The total number of URLs in the output is thus slightly
lower than in the input.
Table 1 shows the prevalence of each format both in termsof URLs that use that format, and in terms of effective toplevel domains (eTLD), sometimes called pay-level domains

(PLD)2 . For computing PLDs, we used the Guava libraryversion 11.0.2. For a small number of URLs we failed todetermine the PLD, e.g. because they contain an IP address instead of a domain name, but we believe this does
not influence the results significantly.
In a second step, we actually extract RDFa data fromthese pages using the Any23 library (version 0.7) as suggested by Bizer et al., and using the same set of extractor
plugins. We use this library with the default configuration
except for setting metadata nesting3 to off, because microformat extraction generates a substantial number of additional triples in the default setting. Before passing the content to Any23, we read the char set of the page from thecontent-type and recode the page content to UTF-8 (we exclude pages where the character set can not be identified).
We also modify each input page that we expect to contain
OGP markup to define the og prefix. Without this, much of
OGP data would not be extracted by Any23’s RDFa parser
and there is also no specific extractor for OGP data. To
speed up the process of extraction, we exclude some extreme
cases: webpages larger than 3 MB, pages , pages containing more than 200 VCard objects, and also pages where theresult of the extraction exceeds 64 MB. We write the data
in a quintet format: subject, predicate, object, context andthe name of the extractor that produced that quad.
To read the data, we use the same NxParser library that
we use to write the data. Unfortunately, there are invalid
lines in the output that we are not able to read back (various exceptions reported by NxParser). Further, some input lines cause the parsing to enter an infinite loop. As atemporary measure until we find the source of these bugs,
we run the parser in a separate thread and terminate thisthread after 500ms. We also limit the size of each input line
to 5KB and do not even attempt to parse lines longer thanthat. Due to these problems, we loose some data: the output
contains 671,454,122 URLs compared to 973,539,519 URLs
that we would expect to contain some data based on regularexpressions. In total, we extract 17,443,606,947 triples. Tables reftbl:topsites-rdfa and 3 and 4 show the top 10 sites asmeasured by the number of triples using RDFa, microdata,or hcard, respectively. The number of triples is an aggregate that reflects both the number of indexed pages in the
crawl (a proxy for the importance of the domain) and theamount of data published per page. Again, we note thatthese lists are not exclusive. For example, youtube.com uses
both microformats, microdata and RDFa within the same
pages.
In terms of vocabulary usage, we show the most commonlyused namespaces in RDFa data in Table 5. We also show themost frequently used classes in terms of the number of URLs
and PLDs in Table 6 and Table 7, respectively. We omit
the http protocol identifier, because all namespaces startwith this protocol identifier, except for a facebook namespace that appears with both http and https. The first tableconfirms that the vast majority of RDFa data on the Web isdue to Facebook’s OGP markup. Unfortunately, OGP does
not always conform with the letter and intent of RDFa. For
example, type information in OGP is given using the og:typepredicate, and not the RDF built-in rdf:type predicate. This
explains the difference between Table 5 vs Table 6 and Ta2
3

http://en.wikipedia.org/wiki/Public_Suffix_List
any23.extraction.metadata.nestingSite
facebook.com
tabelog.com
venere.com
yahoo.com
tripadvisor.co.uk
tripadvisor.it
tripadvisor.com
tripadvisor.fr
tripadvisor.jp
tripadvisor.es
tripadvisor.deanswers.com
myspace.com
tripadvisor.in
daodao.com
tripadvisor.com.tw
tripadvisor.ru
imdb.com
youtube.com
bestbuy.com

Triple count
1,739,664,342
662,028,717
366,531,732
223,125,828
195,314,434
183,603,052
179,970,956
134,442,146
125,976,435
124,845,123
96,635,499
86,721,016
79,984,056
69,763,161
66,014,882
63,430,680
41,199,304
40,537,631
39,942,197
35,910,433

Table 2: Top sites by number of triples, RDFa onlySite
myspace.com
yelp.combbb.org
imdb.com
thefreelibrary.com
powells.com
youtube.com
homefinder.com
reverbnation.com
kino-teatr.ru
eventful.com
cylex.degoodreads.com
bandcamp.com
bizrate.com
businesswire.com
wat.tvavvo.combarnesandnoble.com
patch.com

Triple count133,287,800
94,149,823
85,225,323
37,925,513
37,208,120
31,056,409
26,299,315
25,118,391
20,331,369
15,550,954
15,078,003
14,288,282
12,484,280
11,372,475
10,716,450
9,488,095
9,280,173
9,113,367
8,444,559
8,157,515

Table 3: Top sites by number of triples, microdata
onlyFormat
RDFa
OGP
microdata
microformat
XFN
no dataAbs URL
795,081,604
711,747,491
226,913,004
272,470,501
35,344,618
2,196,204,478

Pct URL
25.08 %
22.45 %
7.16 %
8.60 %
4.27 %
69.29 %

Abs PLD
1,306,827
1,140,880
93,463
1,755,733
1,700,377
30,809,476

Pct PLD
4.04%
3.53%
0.29%
5.43%
5.26%
95.27%

Table 1: Results from pattern-based analysis NU RL = 3, 169, 743, 997, NP LD = 32, 339, 522
Site
yahoo.com
twitter.com
linkedin.com
yellowpages.com
tvtrip.com
youtube.com
myspace.com
nii.ac.jp
nj.compatch.com
chow.com
minecraftforum.net
oregonlive.com
everycarlisted.com
nydailynews.com
last.fm
citysearch.com
washingtonpost.com
nieuwsblad.becleveland.com

Triple count572,687,378
534,336,425
252,481,792
97,624,187
53,746,582
43,330,641
41,110,226
40,752,988
38,202,997
38,003,049
37,705,040
35,891,626
33,159,011
32,75,0040
32,211,122
30,302,919
28,444,466
27,926,328
27,497,607
26,998,847

Table 4: Top sites by number of triples, hcard onlyble 7: most OGP data does not define instances of any RDF
class. As already mentioned above, most users of OGP also
ignore the declaration of the og prefix (a problem we deal
with in the extraction) and we can also see a number of variations to the current standard namespace (a problem we havenot dealt with). Further, OGP assigns additional meaning
to the RDFa syntax that is not reflected in the RDFa standard. As an example, the order in which triples are written
on the page matters in OGP, but not in RDFa. For all thesereasons, we believe that Any23 should be extended with a
specific processor for OGP markup that is able to deal withthese peculiarities.
Besides OGP, a smaller amount of data can be attributed
to efforts by Google’s Rich Snippet program and Yahoo’s
retired SearchMonkey program. Social markup in the formof FOAF and SIOC is also present in a large number of domains as shown in Table 7. The fact that these vocabularies
do not show up as prominently in Table 6 means that they
are used more in the less deeply crawled part of the web.
For microdata, we only list the top namespaces in Table 8 and Table 9, because Any23’s microdata extractor incorporates the class name into the namespace. In microdata, only two vocabularies (schema.org and Google’s datavocabulary.org) have gained significant traction so far, andthe latter is expected to be replaced by the former.
It holds for both RDFa and microdata that the types ofobjects that are marked up is biased by the use case of searchengine optimization, i.e. site owners prefer to mark up datathat is used by the search engines to enrich search result
presentation (e.g reviews, business listings). Schemas forthese types of objects have also existed longer. We also observe a natural preference to mark up simple types of objects(e.g. breadcrumbs), though we did not formally investigate
the relationship between the complexity of markup and its
adoption.

4.

CONCLUSIONSWe presented metadata statistics from the analysis of alarge, recent sample of the Web, which has been extracted
from the crawl of a search engine and therefore provides asearch-engine centric view on the Web. Current web searchengines are biased toward authoritative, head sites with valuable textual content, and are not specifically looking for dataon the Web. We expect that a search engine specifically built
for data would give less weight to authority and textual content and perform deeper crawling on sites that provide large
and valuable data, by some measure of quantity and quality.
Nonetheless, our work shows an impressive progress in the
adoption of markup on the Web with over 30% of our collection containing some microformat, RDFa or microdata
markup. Microformats and RDFa are the most popular
choices of syntax. The level of microformats usage seems to
be flat, while RDFa adoption has grown significantly compared to previous studies. This is due almost exclusively to
OGP markup, though there is a variety of usage in the longtail, in particular social vocabularies. On the other hand,the adoption of microdata is driven so far only by the successof schema.org.
There is significant future work to be done in order toevaluate the quality and practical usefulness of data embedded in HTML, with respect to some existing or novel tasks.
In previous work, we have looked at the extent to which embedded metadata could be used to enrich web search results[1], but data on the Web is likely to be useful in a muchbroader array of applications.

5.

REFERENCES[1] K. Haas, P. Mika, P. Tarjan, and R. Blanco. Enhanced
results for web search. In W.-Y. Ma, J.-Y. Nie, R. A.
Baeza-Yates, T.-S. Chua, and W. B. Croft, editors,
SIGIR, pages 725–734. ACM, 2011.

Namespace
ogp.me/ns#
www.facebook.com/2008/
www.w3.org/1999/02/22-rdf-syntax-ns#
rdf.data-vocabulary.org/#
purl.org/dc/terms/
https://www.facebook.com/2008/
mixi-platform.com/ns#
ogp.me/ns/fb#
creativecommons.org/ns#
www.w3.org/2006/vcard/ns#
http://
http://www.facebook.com/
http://www.w3.org/2000/01/rdf-schema#
http://developers.facebook.com/schema/
http://search.yahoo.com/searchmonkey/commerce/
http://purl.org/dc/elements/1.1/
http://opengraphprotocol.org/schema/
http://search.yahoo.com/searchmonkey/media/
http://oexchange.org/spec/0.8/rel/
http://xmlns.com/foaf/0.1/

URLs
493,443,016
150,246,016
26,402,165
19,413,470
16,424,800
7,472,815
6,323,861
4,636,260
4,622,272
4,205,037
3,881,321
3,126,045
3,042,839
2,720,567
2,664,743
2,642,796
2,293,024
2,095,577
2,034,467
1,837,749

Table 5: Top namespaces in RDFa as measured by the number of URLs

Class
rdf.data-vocabulary.org/#Breadcrumbrdf.data-vocabulary.org/#Review-aggregate
rdf.data-vocabulary.org/#Organizationwww.w3.org/2006/vcard/ns#VCard
search.yahoo.com/searchmonkey/commerce/Business
rdf.data-vocabulary.org/#Review
rdf.data-vocabulary.org/#Rating
rdf.data-vocabulary.org/#review-aggregate
xmlns.com/foaf/0.1/Image
search.yahoo.com/searchmonkey/product/Product
http://rdf.data-vocabulary.org/#Address
http://www.purl.org/stuff/rev#Review
http://rdf.data-vocabulary.org/#Product
http://purl.org/goodrelations/v1#UnitPriceSpecification
http://purl.org/goodrelations/v1#Offering
http://xmlns.com/foaf/0.1/Agent
http://xmlns.com/foaf/0.1/Document
http://www.w3.org/2004/02/skos/core#Concept
http://xmlns.com/foaf/0.1/Group
http://rdfs.org/sioc/ns#Item

URLs
11,336,922
5,571,178
3,678,229
2,858,916
2,727,213
1,980,811
1,714,996
1,453,439
1,446,290
1,202,002
1,087,380
746,858
673,079
648,598
599,703
517,089441,694
406,776
369,176
363,308

Table 6: Top classes in RDFa as measured by the number of URLs with at least one instanceClass
xmlns.com/foaf/0.1/Image
xmlns.com/foaf/0.1/Document
rdfs.org/sioc/ns#Item
rdfs.org/sioc/ns#UserAccountwww.w3.org/2004/02/skos/core#Concept
rdf.data-vocabulary.org/#Breadcrumbrdfs.org/sioc/ns#Post
rdf.data-vocabulary.org/#Review-aggregate
rdfs.org/sioc/types#BlogPost
rdfs.org/sioc/types#Comment
http://rdf.data-vocabulary.org/#Rating
http://rdf.data-vocabulary.org/#Organization
http://www.w3.org/2006/vcard/ns#Address
http://purl.org/goodrelations/v1#BusinessEntity
http://purl.org/goodrelations/v1#UnitPriceSpecification
http://rdf.data-vocabulary.org/#Review
http://rdf.data-vocabulary.org/#Product
http://purl.org/goodrelations/v1#QuantitativeValue
http://rdf.data-vocabulary.org/#Address
http://purl.org/goodrelations/v1#Offering

PLDs
30,90325,09019,583
15,058
9,757
5,427
5,342
3,307
2,970
2,695
2,114
1,759
1,655
1,608
1,385
1,294
1,246
1,051
932
787

Table 7: Top classes in RDFa as measured by the number of PLDs with at least one instanceNamespacewww.w3.org/1999/xhtml/microdata#
www.w3.org/1999/02/22-rdf-syntax-ns#
purl.org/dc/terms/
data-vocabulary.org/Breadcrumb/
schema.org/MusicGroup/
schema.org/MusicRecording/
schema.org/Person/
schema.org/Product/schema.org/VideoObject/
http://schema.org/Article/
http://schema.org/WebPage/
http://data-vocabulary.org/Product/
http://schema.org/PostalAddress/
http://schema.org/Offer/
http://data-vocabulary.org/Review-aggregate/
http://schema.org/AggregateRating/
http://schema.org/LocalBusiness/
http://schema.org/Organization/
http://data-vocabulary.org/Offer/
http://schema.org/Review/

URLs
67,087,467
66,745,726
46,675,266
19,368,347
6,699,903
6,591,236
4,650,659
3,667,023
3,228,156
3,052,457
2,928,410
2,742,977
2,736,213
2,553,617
2,152,533
2,048,232
2,043,005
1,640,501
1,628,027
1,281,548

Table 8: Top namespaces in microdata as measured by the number of URLs

Namespacedata-vocabulary.org/Breadcrumb
schema.org/PostalAddress
schema.org/LocalBusiness
schema.org/Productdata-vocabulary.org/Organization
schema.org/Offer
schema.org/Organization
data-vocabulary.org/Address
schema.org/Article
schema.org/MusicGroup
http://schema.org/MusicAlbum
http://www.schema.org/MusicRecording
http://schema.org/Person
http://data-vocabulary.org/Product
http://data-vocabulary.org/Review-aggregate
http://schema.org/AggregateRating
http://schema.org/WebPage
http://data-vocabulary.org/Rating
http://schema.org/GeoCoordinates
http://schema.org/Place

PLDs
14,623
11,476
8,820
6,817
3,765
3,654
3,614
3,529
3,283
3,253
2,974
2,941
2,676
2,596
2,450
2,380
2,132
1,947
1,651
1,634

Table 9: Top namespaces in microdata as measured by the number of PLDs