Repository: horovod/horovod

Issue:
Getting error while running Horovod for spark clusters which dosen't have GPU's or NCCL
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet): MKL enabled Tensorflow
2. Framework version: 1.13.1
3. Horovod version: 0.16.1
4. MPI version: 4.0.0
5. CUDA version: NA
6. NCCL version: NA
7. Python version: 3.6.6
8. OS and version: CentOS 7

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.md)?
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.md)?

**Your question:**
I'm trying to follow [this link](https://github.com/horovod/horovod/blob/master/docs/spark.md), and getting below error while executing the ``python keras_spark_rossmann.py`` script.

```
Traceback (most recent call last):
  File "keras_spark_rossmann.py", line 266, in <module>
    categorical_cols)
  File "keras_spark_rossmann.py", line 194, in build_vocabulary
    vocab[col] = sorted([r[0] for r in df.select(col).distinct().collect()])
TypeError: '<' not supported between instances of 'NoneType' and 'str'
```

Also I'm getting below errors when executing the commands on pyspark shell.

```
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job 0 cancelled part of cancelled job group horovod.spark.run.0
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887)
	at org.apache.spark.scheduler.DAGScheduler.handleJobCancellation(DAGScheduler.scala:1822)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleJobGroupCancelled$1.apply$mcVI$sp(DAGScheduler.scala:906)

```
And this error
```
raise Exception('mpirun exited with code %d, see the error above.' % exit_code)
Exception: mpirun exited with code 255, see the error above.
>>> 2019-03-26 17:11:45 WARN  PythonRunner:66 - Incomplete task 1.1 in stage 0 (TID 1) interrupted: Attempting to kill Python Worker
2019-03-26 17:11:45 WARN  PythonRunner:66 - Incomplete task 2.2 in stage 0 (TID 2) interrupted: Attempting to kill Python Worker
2019-03-26 17:11:45 WARN  PythonRunner:66 - Incomplete task 7.7 in stage 0 (TID 7) interrupted: Attempting to kill Python Worker
2019-03-26 17:11:45 WARN  PythonRunner:66 - Incomplete task 3.3 in stage 0 (TID 3) interrupted: Attempting to kill Python Worker

```


Discussion:
@umangkeshri, this seems like a data issue.  It needs a bit of further debugging.  Can you check which column causes the issue?
I was able to reproduce the error, it happens because of the null values in `PromoInterval` column.  Python 3 doesn't like to sort lists that include None values.

Replacing `build_vocabulary()` with the following definition solves the problem:
```python
def build_vocabulary(df, cols):
    vocab = {}
    for col in cols:
        values = [r[0] for r in df.select(col).distinct().collect()]
        col_type = type([x for x in values if x is not None][0])
        default_value = col_type()
        vocab[col] = sorted(values, key=lambda x: x or default_value)
    return vocab
```

I will make a PR with a fix shortly.

Resolving patch:
diff --git a/examples/keras_spark_rossmann.py b/examples/keras_spark_rossmann.py
--- a/examples/keras_spark_rossmann.py
+++ b/examples/keras_spark_rossmann.py
@@ -189,7 +189,10 @@ def prepare_df(df):
 def build_vocabulary(df, cols):
     vocab = {}
     for col in cols:
-        vocab[col] = sorted([r[0] for r in df.select(col).distinct().collect()])
+        values = [r[0] for r in df.select(col).distinct().collect()]
+        col_type = type([x for x in values if x is not None][0])
+        default_value = col_type()
+        vocab[col] = sorted(values, key=lambda x: x or default_value)
     return vocab
 
 


Test patch: