Package: homepage2vec
Homepage2VecHomepage2Vec: Language-Agnostic Website Embedding and Classification based on Curlie labelshttps://arxiv.org/pdf/2201.03677.pdfPublished @ International AAAI Conference on Web and Social Media (ICWSM)Homepage2Vec is a pre-trained model that supports the classification and embedding of websites starting from their homepage.Left:Projection in two dimensions with t-SNE of the embedding of 5K random samples of the testing set. Colors represent the 14 classes.Right:The projection with t-SNE of some popular websites shows that embedding vectors effectively capture website topics.Curated Curlie DatasetWe release the full training dataset obtained from Curlie. The dataset includes the websites (online in April 2021) with the URL recognized as homepage, and it contains the original labels, the labels aligned to English, and the fetched HTML pages.Get it here:https://doi.org/10.6084/m9.figshare.19406693Getting started with the libraryInstallation:Install the library with pip:pip install homepage2vecNote: Despite being optimized for GPUs, you can restrict the installation to a CPU-only usage with the flag--extra-index-url https://download.pytorch.org/whl/cpuduring the pip installation.Usage:importloggingfromhomepage2vec.modelimportWebsiteClassifierlogging.getLogger().setLevel(logging.DEBUG)model=WebsiteClassifier()website=model.fetch_website('epfl.ch')scores,embeddings=model.predict(website)print("Classes probabilities:",scores)print("Embedding:",embeddings)Or if you have the HTML document already downloaded:website=Webpage("<url>")website.html="..."website.screenshot_path="..."# optionalscores,embeddings=model.predict(website)Result:Classes probabilities: {'Arts': 0.3674524128437042, 'Business': 0.0720655769109726,
 'Computers': 0.03488553315401077, 'Games': 7.529282356699696e-06, 
 'Health': 0.02021787129342556, 'Home': 0.0005890956381335855, 
 'Kids_and_Teens': 0.3113572597503662, 'News': 0.0079914266243577, 
 'Recreation': 0.00835705827921629, 'Reference': 0.931416392326355, 
 'Science': 0.959597110748291, 'Shopping': 0.0010162043618038297, 
 'Society': 0.23374591767787933, 'Sports': 0.00014659571752417833}
 
Embedding: [-4.596550941467285, 1.0690114498138428, 2.1633379459381104,
 0.1665923148393631, -4.605356216430664, -2.894961357116699, 0.5615459084510803, 
 1.6420538425445557, -1.918184757232666, 1.227172613143921, 0.4358430504798889, 
 ...]The library automatically downloads the pre-trained models homepage2vec and XLM-R at the first usage.Using visual featuresIf you wish to use the prediction using the visual features, Homepage2vec needs to take a screenshot of the website. This means you need a working copy of Selenium and the Chrome browser. Please note that as reported in the reference paper, the performance improvement is limited.Install the visual package of homepage2vec withpip install homepage2vec[visual].Install theSelenium Chrome web driver, and add the folder to the system $PATH variable.You need a local copy of Chrome browser (SeeGetting started).Getting involvedWe invite contributions to Homepage2Vec! Please open apull requestif you have any suggestions.Original publicationHomepage2Vec: Language-Agnostic Website Embedding and ClassificationSylvain Lugeon, Tiziano Piccardi, Robert WestCurrently, publicly available models for website classification do not offer an embedding method and have limited support for languages beyond English. We release a dataset of more than two million category-labeled websites in 92 languages collected from Curlie, the largest multilingual human-edited Web directory. The dataset contains 14 website categories aligned across languages. Alongside it, we introduce Homepage2Vec, a machine-learned pre-trained model for classifying and embedding websites based on their homepage in a language-agnostic way. Homepage2Vec, thanks to its feature set (textual content, metadata tags, and visual attributes) and recent progress in natural language representation, is language-independent by design and generates embedding-based representations. We show that Homepage2Vec correctly classifies websites with a macro-averaged F1-score of 0.90, with stable performance across low- as well as high-resource languages. Feature analysis shows that a small subset of efficiently computable features suffices to achieve high performance even with limited computational resources. We make publicly available the curated Curlie dataset aligned across languages, the pre-trained Homepage2Vec model, and libraries.https://arxiv.org/pdf/2201.03677.pdfCitation formatIf you found the provided resources useful, please cite the paper:BibTex@inproceedings{lugeon2022language,
  title={Homepage2Vec: Language-Agnostic Website Embedding and Classification},
  author={Lugeon, Sylvain and Piccardi, Tiziano and West, Robert},
  booktitle={Proceedings of the International AAAI Conference on Web and Social Media},
  year={2022}
}ChicagoLugeon, Sylvain, Tiziano Piccardi, and Robert West. "Homepage2Vec: Language-Agnostic Website Embedding and Classification." In Proceedings of the International AAAI Conference on Web and Social Media, 2022Dataset LicenseCreative Commons Attribution 3.0 Unported License -CurlieLearn more how to contribute:https://curlie.org/docs/en/about.html