Repository: RasaHQ/rasa
Pull request: Return default output when parsing an empty text
fixes #240

Problem statement:
Properly handle empty strings when passed to parse
 
version: rasa-nlu==0.7.0a8

```
INFO:root:using mitie backend
INFO:root:Started http server on port 5000

root@177-Rancher-host:/home/devuser/rasa_nlu-master# Error categorizing text:
[2017-03-27 05:02:00,849] ERROR in app: Exception on /parse [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "build/bdist.linux-x86_64/egg/rasa_nlu/server.py", line 42, in decorated
  File "build/bdist.linux-x86_64/egg/rasa_nlu/server.py", line 63, in parse_get
  File "build/bdist.linux-x86_64/egg/rasa_nlu/data_router.py", line 189, in parse
    response = model.interpreter.parse(data['text'])
  File "build/bdist.linux-x86_64/egg/rasa_nlu/interpreters/mitie_interpreter.py", line 60, in parse
    intent, score = self.get_intent(tokens)
  File "build/bdist.linux-x86_64/egg/rasa_nlu/interpreters/mitie_interpreter.py", line 53, in get_intent
    label, score = self.classifier(tokens, self.featurizer.feature_extractor)
  File "/usr/local/lib/python2.7/dist-packages/mitie/mitie.py", line 747, in __call__
    raise Exception("Unable to classify text.")
Exception: Unable to classify text.
10.236.220.173 - - [2017-03-27 05:02:00] "POST /parse HTTP/1.1" 500 412 0.006278


 ************************** FATAL ERROR DETECTED **************************
 ************************** FATAL ERROR DETECTED **************************
 ************************** FATAL ERROR DETECTED **************************

Two fatal errors have been detected, the first was inappropriately ignored.
To prevent further fatal errors from being ignored this application will be
terminated immediately and you should go fix this buggy program.

The error message from this fatal error was:


Error detected at line 43.
Error detected in file src/text_feature_extraction.cpp.
Error detected in function mitie::text_sample_type mitie::extract_BoW_features(const std::vector<std::__cxx11::basic_string<char>                                       >&).

Failing expression was words.size() > 0.
words can't be empty



python: ../dlib/dlib/vectorstream/../error.h:268: void dlib::fatal_error::check_for_previous_fatal_errors(): Assertion `false' failed.
```

Hints from issue discussion:
@NagarajGond what is the text you are sending to the API endpoint that triggers this behavior?
sorry for the late reply. 
It took me hours figure out that my code was sending empty string to  RASA sever.
No worries, it's a bug that needs fixing.

Commit message: added test to check if empty parsing works

Patch to review:
diff --git a/_pytest/test_server.py b/_pytest/test_server.py
index c97b29c78938..7e60dcb87b8c 100644
--- a/_pytest/test_server.py
+++ b/_pytest/test_server.py
@@ -54,6 +54,10 @@ def test_status(client):
         u"/parse?q=hello ńöñàśçií",
         [{u"entities": {}, u"confidence": 1.0, u"intent": u"greet", u"_text": u"hello ńöñàśçií"}]
     ),
+    ResponseTest(
+        u"/parse?q=",
+        [{u"entities": {}, u"confidence": 0.0, u"intent": None, u"_text": u""}]
+    ),
 ])
 def test_get_parse(client, response_test):
     response = client.get(response_test.endpoint)
diff --git a/rasa_nlu/model.py b/rasa_nlu/model.py
index e35b6a56e213..fee203720dbe 100644
--- a/rasa_nlu/model.py
+++ b/rasa_nlu/model.py
@@ -213,7 +213,7 @@ class Interpreter(object):
     """Use a trained pipeline of components to parse text messages"""
 
     # Defines all attributes (and their default values) that will be returned by `parse`
-    default_output_attributes = {"intent": None, "entities": [], "text": ""}
+    default_output_attributes = {"intent": {"name": None, "confidence": 0.0}, "entities": [], "text": ""}
 
     @staticmethod
     def load(meta, rasa_config):
@@ -254,6 +254,12 @@ def parse(self, text):
         # type: (basestring) -> dict
         """Parse the input text, classify it and return an object containing its intent and entities."""
 
+        if not text:
+            # Not all components are able to handle empty strings. So we need to prevent that...
+            # This default return will not contain all output attributes of all components,
+            # but in the end, no one should pass an empty string in the first place.
+            return self.default_output_attributes.copy()
+
         current_context = self.context.copy()
 
         current_context.update({


Code review comments:
diff_hunk: @@ -54,6 +54,10 @@ def test_status(client):
         u"/parse?q=hello ńöñàśçií",
         [{u"entities": {}, u"confidence": 1.0, u"intent": u"greet", u"_text": u"hello ńöñàśçií"}]
     ),
+    ResponseTest(
+        u"/parse?q=",
+        [{u"entities": {}, u"confidence": 0.0, u"intent": None, u"_text": u""}]
path: _pytest/test_server.py
text: @user1:
Just saw this, since we import `from __future__ import unicode_literals` we don't need all the `u` prefixes anymore. Different Issue?

diff_hunk: @@ -213,7 +213,7 @@ class Interpreter(object):
     """Use a trained pipeline of components to parse text messages"""
 
     # Defines all attributes (and their default values) that will be returned by `parse`
-    default_output_attributes = {"intent": None, "entities": [], "text": ""}
+    default_output_attributes = {"intent": {"name": None, "confidence": 0.0}, "entities": [], "text": ""}
path: rasa_nlu/model.py
text: @user1:
this is translated to `{"entities": [], "intent": {"confidence": 0.0, "name": null}, "text": ""}` by json.dumps. I am wondering if an empty string would be better than `null`?