Revert "[HOT FIX] [YARN] Check whether `/lib` exists before listing its files"

This reverts commit 38cb2c3a36a5c9ead4494cbc3dde008c2f0698ce.
Revert "SPARK-2624 add datanucleus jars to the container in yarn-cluster"

This reverts commit a975dc32799bb8a14f9e1c76defaaa7cfbaf8b53.
[SPARK-4753][SQL] Use catalyst for partition pruning in newParquet.

Author: Michael Armbrust <michael@databricks.com>

Closes #3613 from marmbrus/parquetPartitionPruning and squashes the following commits:

4f138f8 [Michael Armbrust] Use catalyst for partition pruning in newParquet.

(cherry picked from commit f5801e813f3c2573ebaf1af839341489ddd3ec78)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>