cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.15468/dl.927xkd
title: Occurrence Download

description: A dataset containing 1845651 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is one of (Calidris alba (Pallas, 1764))"
 ]
}

 The dataset includes 1845651 records from 833 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0108490-240626123714530/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.m3fm7r
title: Occurrence Download

description: A dataset containing 1153180 species occurrences available in GBIF matching the query:
{
 "and" : [
 "Country is Indonesia",
 "DatasetKey is EOD – eBird Observation Dataset",
 "HasCoordinate is true",
 "HasGeospatialIssue is false",
 "OccurrenceStatus is Present"
 ]
}

 The dataset includes 1153180 records from 1 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0021964-250402121839773/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.hyf7xs
title: Occurrence Download

description: A dataset containing 28512 species occurrences available in GBIF matching the query:
{
 "and" : [
 "Continent is North America",
 "Country is United States of America",
 "HasCoordinate is true",
 "HasGeospatialIssue is false",
 "OccurrenceStatus is Present",
 "TaxonKey is Robinia pseudoacacia L."
 ]
}

 The dataset includes 28512 records from 190 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0021978-250402121839773/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.9hzrsn
title: Occurrence Download

description: A dataset containing 1828565 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is Aix sponsa (Linnaeus, 1758)"
 ]
}

 The dataset includes 1828565 records from 225 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0036400-190918142434337/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.bff7j8
title: Occurrence Download

description: A dataset containing 267 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is Lethrinus haematopterus Temminck & Schlegel, 1844"
 ]
}

 The dataset includes 267 records from 41 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0052053-231120084113126/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.t3hyiu
title: Occurrence Download

description: A dataset containing 5629 species occurrences available in GBIF matching the query:
{
 "and" : [
 {
 "or" : [
 "TaxonKey is Platyrrhinus oblongus",
 "TaxonKey is Platyrrhinus lineatus nigellus (Gardner & Carter, 1972)",
 "TaxonKey is Platyrrhinus recifinus (Thomas, 1901)",
 "TaxonKey is Platyrrhinus dorsalis (Thomas, 1900)",
 "TaxonKey is Platyrrhinus helleri (Peters, 1866)",
 "TaxonKey is Platyrrhinus vittatus (Peters, 1860)",
 "TaxonKey is Platyrrhinus brachycephalus (Rouk & Carter, 1972)",
 "TaxonKey is Platyrrhinus lineatus (E.Geoffroy, 1810)",
 "TaxonKey is Platyrrhinus aurarius (Handley & Ferris, 1972)",
 "TaxonKey is Platyrrhinus infuscus (Peters, 1880)"
 ]
 },
 "HasCoordinate is true"
 ]
}

 The dataset includes 5629 records from 71 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0002564-160526112335914/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.s04adu
title: Occurrence Download

description: A dataset containing 2351083 species occurrences available in GBIF matching the query:
{
 "and" : [
 "Geometry POLYGON((96.94336 1.40611,106.43555 1.40611,106.43555 28.76766,96.94336 28.76766,96.94336 1.40611))",
 "HasCoordinate is true",
 "TaxonKey is Aves",
 "HasGeospatialIssue is false"
 ]
}

 The dataset includes 2351083 records from 63 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0036416-190918142434337/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.uih9mg
title: Occurrence Download

description: A dataset containing 12511 species occurrences available in GBIF matching the query:
{
 "and" : [
 "TaxonKey is Chrysophyllum L.",
 "HasCoordinate is true",
 "HasGeospatialIssue is false"
 ]
}

 The dataset includes 12511 records from 262 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0002564-170627171947987/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.dxgze5
title: Occurrence Download

description: A dataset containing 265990 species occurrences available in GBIF matching the query:
{
 "and" : [
 {
 "not" : {
 "PublishingCountry" : [
 "is Norway"
 ]
 }
 },
 "Country is Svalbard and Jan Mayen"
 ]
}

 The dataset includes 265990 records from 551 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0046692-210914110416597/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.wceg2y
title: Occurrence Download

description: A dataset containing 491 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is Cordulegaster diadema Selys, 1868"
 ]
}

 The dataset includes 491 records from 25 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0045160-231120084113126/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.vefze1
title: Occurrence Download

description: A dataset containing 1990 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is Tulipa agenensis Redouté"
 ]
}

 The dataset includes 1990 records from 55 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0002564-171020152545675/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.9kza7a
title: Occurrence Download

description: A dataset containing 962857 species occurrences available in GBIF matching the query:
{
 "and" : [
 "HasCoordinate is true",
 "HasGeospatialIssue is false",
 {
 "or" : [
 "Month is May",
 "Month is June",
 "Month is April",
 "Month is July",
 "Month is August",
 "Month is September"
 ]
 },
 "OccurrenceStatus is Present",
 "TaxonKey is Emberiza citrinella Linnaeus, 1758"
 ]
}

 The dataset includes 962857 records from 898 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0063827-210914110416597/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.iln4qy
title: Occurrence Download

description: A dataset containing 550 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is Oligoryzomys chacoensis (Myers & Carleton, 1981)"
 ]
}

 The dataset includes 550 records from 16 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0010105-190415153152247/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.4utgu1
title: Occurrence Download

description: A dataset containing 158252 species occurrences available in GBIF matching the query:
{
 "DatasetKey" : [
 "is Bioversity Collecting Mission Database"
 ]
}

 The dataset includes 158252 records from 1 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0007150-180508205500799/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.jec2bf
title: Occurrence Download

description: A dataset containing 275 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is Boea hygrometrica (Bunge) R.Br."
 ]
}

 The dataset includes 275 records from 22 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0398142-210914110416597/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.4k9zqh
title: Occurrence Download

description: A dataset containing 81 species occurrences available in GBIF matching the query:
{
 "and" : [
 "BasisOfRecord is Specimen",
 "HasCoordinate is true",
 "HasGeospatialIssue is false",
 "TaxonKey is Monodon monoceros Linnaeus, 1758"
 ]
}

 The dataset includes 81 records from 14 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0242335-200613084148143/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.bu653u
title: Occurrence Download

description: A dataset containing 122 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is Blanus strauchi (Bedriaga, 1884)"
 ]
}

 The dataset includes 122 records from 23 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0010105-200613084148143/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.hgkz46
title: Occurrence Download

description: A dataset containing 372 species occurrences available in GBIF matching the query:
{
 "and" : [
 "Country is Mexico",
 "TaxonKey is Abies religiosa (Kunth) Schltdl. & Cham.",
 "Year 2010-2021"
 ]
}

 The dataset includes 372 records from 8 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0287700-200613084148143/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.0ynmaf
title: Occurrence Download

description: A dataset containing 52 species occurrences available in GBIF matching the query:
{
 "and" : [
 "BasisOfRecord is Human Observation",
 {
 "or" : [
 "Country is Mexico",
 "Country is United States of America"
 ]
 },
 "HasCoordinate is true",
 "TaxonKey is Holbrookia approximans Baird, 1859",
 "HasGeospatialIssue is false"
 ]
}

 The dataset includes 52 records from 3 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0007150-181003121212138/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.jjbuyy
title: Occurrence Download

description: A dataset containing 15 species occurrences available in GBIF matching the query:
{
 "and" : [
 "TaxonKey is Lagotis crassifolia Prain",
 "HasCoordinate is true",
 "HasGeospatialIssue is false"
 ]
}

 The dataset includes 15 records from 2 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0320239-200613084148143/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.nq7k37
title: Occurrence Download

description: A dataset containing 20665 species occurrences available in GBIF matching the query:
{
 "and" : [
 "BasisOfRecord is Human Observation",
 "Country is United Kingdom of Great Britain and Northern Ireland",
 "OccurrenceStatus is Present",
 "TaxonKey is Myriophyllum L."
 ]
}

 The dataset includes 20665 records from 116 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0242317-200613084148143/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.axqca3
title: Occurrence Download

description: A dataset containing 907 species occurrences available in GBIF matching the query:
{
 "DatasetKey" : [
 "is (Appendix 2-8) Distribution of planktonic foraminifera in late Neogene sediments of DSDP Site 86-577 in the western Pacific Ocean"
 ]
}

 The dataset includes 907 records from 1 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0010108-160910150852091/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.rzb4ej
title: Occurrence Download

description: A dataset containing 552 species occurrences available in GBIF matching the query:
{
 "and" : [
 "TaxonKey is Protozoa",
 "DecimalLatitude is greater than 38.6508426271732",
 "DecimalLatitude is less than 39.2343733909135",
 "DecimalLongitude is greater than -77.4447931357568",
 "DecimalLongitude is less than -76.6149738274161"
 ]
}

 The dataset includes 552 records from 8 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0007151-180824113759888/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.pzqrqm
title: Occurrence Download

description: A dataset containing 1368 species occurrences available in GBIF matching the query:
{
 "and" : [
 "BasisOfRecord is Human Observation",
 {
 "or" : [
 "Country is China",
 "Country is Chinese Taipei",
 "Country is Hong Kong"
 ]
 },
 "HasCoordinate is true",
 "HasGeospatialIssue is false",
 "TaxonKey is Tringa guttifer (Nordmann, 1835)"
 ]
}

 The dataset includes 1368 records from 7 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0245108-200613084148143/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.qxpr4r
title: Occurrence Download

description: A dataset listing the 900706 species recorded in GBIF matching the query:
{
 "TaxonKey" : [
 "is Tracheophyta"
 ]
}

 The dataset's 900706 records were derived from 5367 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0314170-200613084148143/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.25549/pcra-c14-56531
title: Redemption tidings, vol. 37, nos. 1-52, 1 Jan. - 22 Dec. 1961

description: [description english/roman)] Issued weekly. "A full gospel magazine" -- subtitle.
DOI: 10.15468/dl.t3wjdc
title: Occurrence Download

description: A dataset containing 215 species occurrences available in GBIF matching the query:
{
 "and" : [
 "VerbatimScientificName is Monarcha richardsii",
 "BasisOfRecord is one of (Machine Observation, Human Observation)",
 "Year 2011-2021",
 "HasCoordinate is true",
 "OccurrenceStatus is Present",
 {
 "not" : {
 "BasisOfRecord" : [
 "is one of (Fossil, Living Specimen, Specimen, Material citation, Material sample)"
 ]
 }
 }
 ]
}

 The dataset includes 215 records from 4 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0092065-230530130749713/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.66djap
title: Occurrence Download

description: A dataset containing 0 species occurrences available in GBIF matching the query:
{
 "and" : [
 "VerbatimScientificName is Monarcha megarhynchus",
 "BasisOfRecord is one of (Machine Observation, Human Observation)",
 "Year 2011-2021",
 "HasCoordinate is true",
 "OccurrenceStatus is Present",
 {
 "not" : {
 "BasisOfRecord" : [
 "is one of (Fossil, Living Specimen, Specimen, Material citation, Material sample)"
 ]
 }
 }
 ]
}

 The dataset includes 0 records from 0 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0092059-230530130749713/datasets/export for details.
DOI: 10.15468/dl.yeua8s
title: Occurrence Download

description: A dataset containing 399 species occurrences available in GBIF matching the query:
{
 "and" : [
 {
 "or" : [
 "BasisOfRecord is Machine Observation",
 "BasisOfRecord is Specimen",
 "BasisOfRecord is Material sample"
 ]
 },
 "HasCoordinate is true",
 "TaxonKey is Tamandua tetradactyla (Linnaeus, 1758)",
 "HasGeospatialIssue is false"
 ]
}

 The dataset includes 399 records from 44 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0051025-200221144449610/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.25549/pcra-c14-56493
title: Redemption tidings, vol. 37, nos. 1-52, 1 Jan. - 22 Dec. 1961

description: [description english/roman)] Issued weekly. "A full gospel magazine" -- subtitle.
DOI: 10.15468/dl.7zq72a
title: Occurrence Download

description: A dataset containing 138 species occurrences available in GBIF matching the query:
{
 "and" : [
 {
 "or" : [
 "BasisOfRecord is Observation",
 "BasisOfRecord is Machine Observation",
 "BasisOfRecord is Human Observation",
 "BasisOfRecord is Occurrence evidence"
 ]
 },
 "Country is China",
 "HasCoordinate is true",
 "HasGeospatialIssue is false",
 "OccurrenceStatus is Present",
 "TaxonKey is Emberiza koslowi Bianchi, 1904",
 "Year 1972-2022"
 ]
}

 The dataset includes 138 records from 6 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0278807-210914110416597/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.15468/dl.ct89vk
title: Occurrence Download

description: A dataset containing 36891 species occurrences available in GBIF matching the query:
{
 "TaxonKey" : [
 "is Sempervivum tectorum L."
 ]
}

 The dataset includes 36891 records from 489 constituent datasets; see https://api.gbif.org/v1/occurrence/download/0040510-230530130749713/datasets/export for details.
 
Data from some individual datasets included in this download may be licensed under less restrictive terms.
DOI: 10.5281/zenodo.10795434
title: A Comprehensive Dataset for Webpage Classification (Part 2: Benign 1)

description: This dataset, split across three parts due to Zenodo's size constraints, serves as a fundamental resource for enhancing webpage classification techniques. It encompasses 1,069,715 URLs, each annotated with labels to signify their categorization into Malicious, Benign, or Adult content, and further into 20 detailed sublabels for granular analysis. The dataset is designed to facilitate the evaluation and benchmarking of machine learning models, notably Stochastic Gradient Descent (SGD) and Support Vector Classifier (SVC), across a variety of tokenization methods and input types, including URLs, raw HTML, and parsed HTML content.

The primary objective of assembling this dataset is to support research into effective webpage classification, thereby improving content prioritization and filtering in web crawling applications. It has been meticulously curated to provide a robust framework for studying the impact of different feature representation techniques on classification accuracy.

The dataset is structured as JSON lines (jsonl) files, with each entry detailing a URL's label, sublabel, source, status code, and HTML content. This comprehensive dataset is divided into three parts due to size constraints on Zenodo, each targeting specific content categories to ensure ease of use and accessibility for researchers:



Part 1: Adult & Malicious encompasses URLs classified under Adult and Malicious categories, offering insights into content that requires stringent filtering.

Part 2: Benign 1 and Part 3: Benign 2 cover benign URLs, facilitating the study of safe web content and its classification nuances.


We also created a .csv file without the HTML content so it is easier to work with URLs only, this .csv file contains the next columns `['uid', 'url', 'label', 'sublabel']`

By providing this dataset, we aim to contribute significantly to the field of webpage classification, offering a valuable asset for researchers and practitioners looking to advance the state of web crawling technology and its applications.

 

JSON line format for each line:

{"url": "<URL>", "label": "<Label>", "sublabel": "<Sublabel>", "source": "<Source of this URL>", "status_code": <Status Code>, "html": "<HTML Textual Content>"}

Other parts of this dataset:



A Comprehensive Dataset for Webpage Classification (Part 1: Adult & Malicious)

A Comprehensive Dataset for Webpage Classification (Part 3: Benign 2)


 

Citation

if you use this dataset, please cite us:

Al-Maamari, M., Istaiti, M., Zerhoudi, S., Dinzinger, M., Granitzer, M. and Mitrovic, J., A COMPREHENSIVE DATASET FOR WEBPAGE CLASSIFICATION.

https://ca-roll.github.io/downloads/A_Comprehensive_Dataset_for_Webpage_Classification.pdf

 

Granitzer, M., Voigt, S., Fathima, N.A., Golasowski, M., Guetl, C., Hecking, T., Hendriksen, G., Hiemstra, D., Martinovič, J., Mitrović, J. and Mlakar, I., 2023. Impact and development of an Open Web Index for open web search. Journal of the Association for Information Science and Technology.

https://doi.org/10.1002/asi.24818