DOI: 10.5281/zenodo.6483447
title: Web Data Commons (November 2017) Property and Datatype Usage Dataset

description: This is a dataset about the usage of properties and datatypes in the Web Data Commons RDFa, Microdata, Embedded JSON-LD, and Microformats Data Sets (November 2017) based on the Common Crawl November 2017 archive. The dataset has been produced using the RDF Property and Datatype Usage Scanner v2.1.1, which is based on the Apache Jena framework. Only RDFa and embedded JSON-LD data were considered, as Microdata and Microformats do not incorporate explicit datatypes. <strong>Dataset Properties</strong> <strong>Size</strong>: 16.2 MiB compressed, 327.9 MiB uncompressed, 1 509 385 rows plus 1 head line determined using <code class="language-bash">gunzip -c measurements.csv.gz | wc -l</code> <strong>Parsing Failures</strong>: The scanner failed to parse 6 024 240<br> triples (~0.11 %) of the source dataset (containing 5 252 606 731 triples). <strong>Content</strong>: CATEGORY: The category (<em>html-embedded-jsonld</em> or <em>html-rdfa</em>) of the Web Data Commons file that has been measured. FILE_URL: The URL of the Web Data Commons file that has been measured. MEASUREMENT: The applied measurement with specific conditions, one of: <strong>UnpreciseRepresentableInDouble</strong>: The number of lexicals that are in the lexical space but not in the value space of <code>xsd:double</code>. <strong>UnpreciseRepresentableInFloat</strong>: The number of lexicals that are in the lexical space but not in the value space of <code>xsd:float</code>. <strong>UsedAsDatatype</strong>: The total number of literals with the datatype. <strong>UsedAsPropertyRange</strong>: The number of statements that specify the datatype as range of the property. <strong>ValidDateNotation</strong>: The number of lexicals that are in the lexical space of <code>xsd:date</code>. <strong>ValidDateTimeNotation</strong>: The number of lexicals that are in the lexical space of <code>xsd:dateTime</code>. <strong>ValidDecimalNotation</strong>: The number of lexicals that represent a number with decimal notation and whose lexical representation is thereby in the lexical space of <code>xsd:decimal</code>, <code>xsd:float</code>, and <code>xsd:double</code>. <strong>ValidExponentialNotation</strong>: The number of lexicals that represent a number with exponential notation and whose lexical representation is thereby in the lexical space of <code>xsd:float</code>, and <code>xsd:double</code>. <strong>ValidInfOrNaNNotation</strong>: The number of lexicals that equals either <code>INF</code>, <code>+INF</code>, <code>-INF</code> or <code>NaN</code> and whose lexical representation is thereby in the lexical space of <code>xsd:float</code>, and <code>xsd:double</code>. <strong>ValidIntegerNotation</strong>: The number of lexicals that represent an integer number and whose lexical representation is thereby in the lexical space of <code>xsd:integer</code>, <code>xsd:decimal</code>, <code>xsd:float</code>, and <code>xsd:double</code>. <strong>ValidTimeNotation</strong>: The number of lexicals that are in the lexical space of <code>xsd:time</code>. <strong>ValidTrueOrFalseNotation</strong>: The number of lexicals that equal either <code>true</code> or <code>false</code> and whose lexical representation is thereby in the lexical space of <code>xsd:boolean</code>. <strong>ValidZeroOrOneNotation</strong>: The number of lexicals that equal either <code>0</code> or <code>1</code> and whose lexical representation is thereby in the lexical space of <code>xsd:boolean</code>, and <code>xsd:integer</code>, <code>xsd:decimal</code>, <code>xsd:float</code>, and <code>xsd:double</code>. <strong>Note</strong>: Lexical representation of <code>xsd:double</code> values in embedded JSON-LD got normalized to always use exponential notation with up to 16 fractional digits (see related code). Be careful by drawing conclusions from according <strong>Valid…</strong> and <strong>Unprecise…</strong> measures. PROPERTY: The property that has been measured. DATATYPE: The datatype that has been measured. QUANTITY: The count of statements that fulfill the condition specified by the measurement per file, property and datatype. <strong>Preview</strong> <pre><code>"CATEGORY","FILE_URL","MEASUREMENT","PROPERTY","DATATYPE","QUANTITY" "html-rdfa","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-rdfa.nq-00000.gz","UnpreciseRepresentableInDouble","http://search.yahoo.com/searchmonkey/product/identifier","http://search.yahoo.com/searchmonkey-datatype/use/sku","5" "html-rdfa","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-rdfa.nq-00000.gz","UnpreciseRepresentableInDouble","http://purl.org/dc/elements/1.1/title","http://www.w3.org/2001/XMLSchema#string","1" "html-rdfa","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-rdfa.nq-00000.gz","UnpreciseRepresentableInDouble","http://opengraphprotocol.org/schema/longitude","http://www.w3.org/2001/XMLSchema#string","2290" "html-rdfa","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-rdfa.nq-00000.gz","UnpreciseRepresentableInDouble","http://dbpedia.org/ontology/Work/runtime","http://www.w3.org/2001/XMLSchema#string","2" "html-rdfa","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-rdfa.nq-00000.gz","UnpreciseRepresentableInDouble","https://schema.org/number","http://www.w3.org/2001/XMLSchema#string","1" […] "html-embedded-jsonld","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-embedded-jsonld.nq-00609.gz","ValidZeroOrOneNotation","http://schema.org/worstRating","http://www.w3.org/2001/XMLSchema#integer","38" "html-embedded-jsonld","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-embedded-jsonld.nq-00609.gz","ValidZeroOrOneNotation","http://schema.org/price","http://www.w3.org/2001/XMLSchema#integer","1" "html-embedded-jsonld","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-embedded-jsonld.nq-00609.gz","ValidZeroOrOneNotation","http://schema.org/ratingValue","http://www.w3.org/2001/XMLSchema#integer","3" "html-embedded-jsonld","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-embedded-jsonld.nq-00609.gz","ValidZeroOrOneNotation","http://schema.org/width","http://www.w3.org/2001/XMLSchema#integer","1" "html-embedded-jsonld","http://data.dws.informatik.uni-mannheim.de/structureddata/2017-12/quads/dpef.html-embedded-jsonld.nq-00609.gz","ValidZeroOrOneNotation","http://schema.org/pageEnd","http://www.w3.org/2001/XMLSchema#integer","4" </code></pre> <strong>Note</strong>: The data contain malformed IRIs, like "<code>xsd:dateTime</code>" (instead of probably "<code>http://www.w3.org/2001/XMLSchema#dateTime</code>"), which are caused by missing namespace definitions in the original source website. <strong>Reproduce</strong> To reproduce this dataset checkout the RDF Property and Datatype Usage Scanner v2.1.1 and execute: <pre><code class="language-bash">mvn clean package java -jar target/Scanner.jar --category html-rdfa --list http://webdatacommons.org/structureddata/2017-12/files/rdfa.list November2017 java -jar target/Scanner.jar --category html-embedded-jsonld --list http://webdatacommons.org/structureddata/2017-12/files/html-embedded-jsonld.list November2017 ./measure.sh November2017 # Wait until the scan has completed. This will take a few days java -jar target/Scanner.jar --results ./November2017/measurements.csv.gz November2017 </code></pre>