GitHub repository: webis-de/WARC-DL
Primary language: Python
Description: This pipeline allows extracting data from WARC files on a CPU cluster and streaming it to a GPU server, where it is processed.