Package: cc2dataset
cc2datasetEasily convert common crawl to a dataset of caption and document. Image/text Audio/text Video/text, ...Common crawl has5M wat files. They provide links of the web.
This simple tool allows you to process one warc in about 50s and get documents link along with the alt text.It also runs deduplication against url+text in order to save on output space and speed up the process.This makes it possible to do the first step of building a dataset likelaion5Bin 70k cpu core hours. (5*10^6*50/(3600))
That's$2.8kusing aws EC2 (0.04$/core hour)Intended usageThis tool produces a collection of link + caption. It is meant as the stage 1 of creating a dataset. It does deduplication and as minimal as possible filtering (does it look like an url / is the caption non empty).This produces a large quantity of raw data that can then be further filtered by appropriate techniques.
An example of stage 2 can be to estimate the similarity between (link, text) with a model such as CLIP. This may reduce the quantity of data by a factor of up to 100x depending on the chosen threshold.What hardware to pick ?CC is big and located at s3 us east 1, so it makes a lot of sense in term of network to use machines located in the same place.cpu128-dy-c6i-32xlargeinstances are advised. Spark stores the non duplicated first stage in local disk. They should be nvme drive for speed during deduplication. At this first stage, one wat takes about 20MB, so the total (over all workers) space must be more than 20MB times wat count. So for example for the whole CC, that means 100TB. So for example that can fit in 150 instances with 1TB nvme drive each. 150 instances of 128 cores is 19200 cores so the whole processing takes 2h. Less instances with bigger hard drives can work too. It's also a possibility to do the processing in multiple pieces if temporary disk space is an issue by specifying--multipart.Document typeThis tool support extracting several documents from CC:image/text: about 300B after dedupimage/text even with empty text: estimated 1Taudio/text: about 2B after deduptext doc : about 10B after dedupvideo/text: about 2B after dedupThey can be selected with eg--document_type audio.
You may experiment with more document kinds by runningpython example single_warc_example.pyand exploring the resulting output.parquet.Installpip install cc2datasetPython examplesCheckout these examples:run_on_spark.pyit shows how to bring your own spark sessionIf you have a slurm cluster, refer tohttps://gist.github.com/rom1504/67ada3dedbecc113ae2dbdfd9c642d83to start a spark cluster there.APIThis module exposes a single functioncc2datasetwhich takes the same arguments as the command line tool:output_paththe output path, should probably start with s3://. The output will be written to this path sufixed by the date (required)wat_index_countthe number of wat index files to read, can be None for all. (default 1)wat_countthe number of wat files to read, can be None for all, will randomly subsample if present. (default 100)masterthe spark master url. (default local)num_coresthe number of cores of each spark executor. (default 128)mem_gbthe memory of each spark executor. (default 256)multipartruns the processing of the specified number of parts, merge at the end (default None)shufflerandomly shuffle the output right before saving (default True)resumethe specific path of the output to resume (default None)spark_buildera function that create a spark session, None will default to the built-in methods (default None)document_typethe kind of document to extract (default image)source_cc_protocolget common crawl from http or s3 (default s3)For developmentEither locally, or ingitpod(doexport PIP_USER=falsethere)Setup a virtualenv:python3 -m venv .env
source .env/bin/activate
pip install -e .to run tests:pip install -r requirements-test.txtthenmake lint
make testYou can usemake blackto reformat the codepython -m pytest -x -s -v tests -k "dummy"to run a specific testThanksVaishaalfor providing the initial CC parsing code with efficient librariesrvencufor optimizing the ccparsing codefor laion5B on which the idea of this package is based on