apache/spark — pull request #2680 by yhuai (2014-10-06 17:53:32+00:00)
Title: [SPARK-3339][SQL] Support for skipping json lines that fail to parse

This PR aims to provide a way to skip/query corrupt JSON records. To do so, we introduce an internal column to hold corrupt JSON records (the default name is `corruptRecord`. This name can be changed by setting the value of `spark.sql.json.columnNameOfCorruptRecord`). When there is a parsing error, we will put the corrupt record in its unparsed format to the internal column. Users can skip/query this column through SQL.

* To query those corrupt records
```
SELECT corruptRecord
FROM jsonTable
WHERE corruptRecord IS NOT NULL
```
* To skip corrupt records and query regular records
```
SELECT field1, field2
FROM jsonTable
WHERE corruptRecord IS NULL
```

Generally, it is not recommended to change the name of the internal column. If the name has to be changed to avoid possible name conflicts, you can use `sqlContext.setConf(SQLConf.COLUMN_NAME_OF_CORRUPT_JSON_RECORD, )` or `sqlContext.sql(SET spark.sql.json.columnNameOfCorruptRecord=)`.