apache/spark — pull request #1536 by ericgarcia (2014-07-22 19:54:57+00:00)
Title: Example pyspark-inputformat for Avro file format

This is an example showing how to map an Avro file to a pyspark RDD.

Starting Pyspark with
`SPARK_CLASSPATH=examples/target/scala-2.10/spark-examples-1.1.0-SNAPSHOT-hadoop1.0.4.jar IPYTHON=1 bin/pyspark`

avroRdd = sc.newAPIHadoopFile("/tmp/data.avro",
"org.apache.avro.mapreduce.AvroKeyInputFormat",
"org.apache.avro.mapred.AvroKey",
"org.apache.hadoop.io.NullWritable",
keyConverter="org.apache.spark.examples.pythonconverters.AvroGenericConverter")

Note that not all data types are implemented.