GitHub repository: kosmylo/articles-retrieval
Primary language: Python
Description: Articles-Retrieval – A Docker Compose stack that scrapes Wikipedia pages, news articles, arXiv papers, and EU government documents on energy topics. It preprocesses articles by removing HTML content, deduplicating based on titles, filtering non-English and corrupted content, and streams clean JSONL data.