Package: commoncrawlzain
Common Crawl PySpark ExamplesThis project provides examples how to process the Common Crawl dataset withApache Sparkand Python:count HTML tagsin Common Crawl's raw response data (WARC files)count web server namesin Common Crawl's metadata (WAT files or WARC files)list host names and correspondingIP addresses(WAT files or WARC files)word count(term and document frequency) in Common Crawl's extracted text (WET files)extract linksfrom WAT files andconstruct the (host-level) web graph– for further details about the web graphs see the projectcc-webgraphwork with thecolumnar URL index(see alsocc-index-tableand notes aboutschema merging):run a SQL query andexport the result as a tableselect WARC records by a SQL query, parse the HTML, extract the text andcount words. Alternatively, the first step (query the columnar index) can be executed using Amazon Athena. The list of WARC record coordinates (CSV or a table created by a CTAS statement) is then passed via--csvor--input_table_format) to the Spark job.Further information about the examples and available options is shown via thecommand-line option--help.Implementing a Custom ExtractorExtending theCCSparkJobisn't difficult and for many use cases it's sufficient to override a single method (process_record). Have a look at one of the examples, eg. tocount HTML tags.SetupTo develop and test locally, you will need to installSpark, see thedetailed instructions, andall required Python modules by runningpip install -r requirements.txt(optionally, and only if you want to query the columnar index)install S3 support librariesso that Spark can load the columnar index from S3Compatibility and RequirementsTested with Spark 2.1.0 – 2.4.6 in combination with Python 2.7 or 3.5, 3.6, 3.7, and with Spark 3.0.0 - 3.2.1 in combination with Python 3.7, 3.8 and 3.9.Get Sample DataTo develop locally, you'll need at least three data files – one for each format used in at least one of the examples. They can be fetched from the following links:warc.gzwat.gzwet.gzAlternatively, runningget-data.shdownloads the sample data. It also writes input files containingsample input asfile://URLsall input of one monthly crawl as relative pathsto use with--input_base_url s3://commoncrawl/resp.--input_base_url https://data.commoncrawl.org/Note that the sample data is from an older crawl (CC-MAIN-2017-13run in March 2017). If you want to use more recent data, please visit theCommon Crawl site.Process Common Crawl Data on SparkRunning locallyFirst, point the environment variableSPARK_HOMEto your Spark installation.
Then submit a job via$SPARK_HOME/bin/spark-submit ./server_count.py \
	--num_output_partitions 1 --log_level WARN \
	./input/test_warc.txt servernamesThis will count web server names sent in HTTP response headers for the sample WARC input and store the resulting counts in the SparkSQL table "servernames" in your warehouse location defined byspark.sql.warehouse.dir(usually in your working directory as./spark-warehouse/servernames).The output table can be accessed via SparkSQL, e.g.,$SPARK_HOME/bin/pyspark
>>> df = sqlContext.read.parquet("spark-warehouse/servernames")
>>> for row in df.sort(df.val.desc()).take(10): print(row)
... 
Row(key=u'Apache', val=9396)
Row(key=u'nginx', val=4339)
Row(key=u'Microsoft-IIS/7.5', val=3635)
Row(key=u'(no server in HTTP header)', val=3188)
Row(key=u'cloudflare-nginx', val=2743)
Row(key=u'Microsoft-IIS/8.5', val=1459)
Row(key=u'Microsoft-IIS/6.0', val=1324)
Row(key=u'GSE', val=886)
Row(key=u'Apache/2.2.15 (CentOS)', val=827)
Row(key=u'Apache-Coyote/1.1', val=790)See alsorunning the Spark shell and submitting Spark jobsPySpark SQL APIRunning in Spark cluster over large amounts of dataAs the Common Crawl dataset lives in the Amazon Public Datasets program, you can access and process it on Amazon AWS (in the us-east-1AWS region) without incurring any transfer costs. The only cost that you incur is the cost of the machines running your Spark cluster.spinning up the Spark cluster:AWS EMRcontains a ready-to-use Spark installation but you'll find multiple descriptions on the web how to deploy Spark on a cheap cluster of AWS spot instances. See alsolaunching Spark on a cluster.choose appropriate cluster-specific settings whensubmitting jobsand also check for relevantcommand-line options(e.g.,--num_input_partitionsor--num_output_partitions, see below)don't forget to deploy all dependencies in the cluster, seeadvanced dependency managementalso the the filesparkcc.pyneeds to be deployed or added as argument--py-files sparkcc.pytospark-submit. Note: some of the examples require further Python files as dependencies.Command-line optionsAll examples show the available command-line options if called with the parameter--helpor-h, e.g.$SPARK_HOME/bin/spark-submit ./server_count.py --helpOverwriting Spark configuration propertiesThere are manySpark configuration propertieswhich allow to tune the job execution or output, see for example seetuning SparkorEMR Spark memory tuning.It's possible to overwrite Spark properties whensubmitting the job:$SPARK_HOME/bin/spark-submit \
    --conf spark.sql.warehouse.dir=myWareHouseDir \
    ... (other Spark options, flags, config properties) \
    ./server_count.py \
    ... (program-specific options)Authenticated S3 Access or Access Via HTTPSince April 2022 there are two ways to access of Common Crawl data:using HTTP/HTTPS and the base URLhttps://data.commoncrawl.org/orhttps://ds5q9oxwqwsfj.cloudfront.net/using the S3 API to read the buckets3://commoncrawl/requires authentication and makes an Amazon Web Services account mandatory.This project cc-pyspark usesboto3to access WARC, WAT or WET files ons3://commoncrawl/. The best way is to ensure that a S3 read-only IAM policy is attached to the the IAM role of the EC2 instances where Common Crawl data is processed, see theIAM user guide. If this is no option (or if the processing is not running on AWS), there are variousoptions to configure credentials in boto3.Installation of S3 Support LibrariesWhile WARC/WAT/WET files are read using boto3, accessing thecolumnar URL index(see option--queryof CCIndexSparkJob) is done directly by the SparkSQL engine and requires that S3 support libraries are available. These libs are usually provided when the Spark job is run on a Hadoop cluster running on AWS (eg. EMR). However, they may not be provided for any Spark distribution and are usually absent when running Spark locally (not in a Hadoop cluster). In these situations, the easiest way is to add the libs as required packages by adding--packages org.apache.hadoop:hadoop-aws:3.2.1to the arguments ofspark-submit. This will makeSpark manage the dependencies- the hadoop-aws package and transitive dependencies are downloaded as Maven dependencies. Note that the required version of hadoop-aws package depends on the Hadoop version bundled with your Spark installation, e.g., Spark 3.2.1 bundled with Hadoop 3.2 (spark-3.2.1-bin-hadoop3.2.tgz).Please also note that:the schema of the URL referencing the columnar index depends on the actual S3 file system implementation: it'ss3://on EMR buts3a://when usings3a.(since April 2022) only authenticated S3 access is possible. This requires that access to S3 is properly set up. For configuration details, seeAuthorizing access to EMRFS data in Amazon S3orHadoop-AWS: Authenticating with S3.Example call to count words in 10 WARC records host under the.istop-level domain:$SPARK_HOME/bin/spark-submit \
    --packages org.apache.hadoop:hadoop-aws:3.3.2 \
    ./cc_index_word_count.py \
    --input_base_url s3://commoncrawl/ \
    --query "SELECT url, warc_filename, warc_record_offset, warc_record_length, content_charset FROM ccindex WHERE crawl = 'CC-MAIN-2020-24' AND subset = 'warc' AND url_host_tld = 'is' LIMIT 10" \
    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
    myccindexwordcountoutput \
    --num_output_partitions 1 \
    --output_format jsonColumnar index and schema mergingThe schema of thecolumnar URL indexhas been extended over time by adding new columns. If you want to query one of the new columns (e.g.,content_languages), the followingSpark configuration optionneeds to be set:--conf spark.sql.parquet.mergeSchema=trueHowever, this option impacts the query performance, so use with care! Please also readcc-index-tableabout configuration options to improve the performance of Spark SQL queries.Alternatively, it's possible configure the table schema explicitly:download thelatest table schema as JSONand use it by adding the command-line argument--table_schema cc-index-schema-flat.json.CreditsExamples are originally ported from Stephen Merity'scc-mrjobwith the following changes and upgrades:based on Apache Spark (instead ofmrjob)boto3supporting multi-part download of data from S3warcioa Python 2 and Python 3 compatible module to access WARC filesFurther inspirations are taken fromcosr-backwritten by Sylvain Zimmer forCommon Search. You definitely should have a look at it if you need a more sophisticated WARC processor (including a HTML parser for example).Mark Litwintschik's blog postAnalysing Petabytes of WebsitesLicenseMIT License, as perLICENSE