[HOT FIX] [YARN] Check whether `/lib` exists before listing its files

This is caused by a975dc32799bb8a14f9e1c76defaaa7cfbaf8b53

Author: Andrew Or <andrew@databricks.com>

Closes #3589 from andrewor14/yarn-hot-fix and squashes the following commits:

a4fad5f [Andrew Or] Check whether lib directory exists before listing its files

(cherry picked from commit 90ec643e9af4c8bbb9000edca08c07afb17939c7)
Signed-off-by: Andrew Or <andrew@databricks.com>