Repository: RasaHQ/rasa
Pull request: Rasa NLU data spec update
**Proposed changes**:

Fix #7063

After some investigation I figured the following:

The support of JSON has not been removed but the provided data needs to be a JSON formatted the way it can be read by `RasaReader`. However, there is no information about it in the spec. For example, I used [this JSON](https://gist.github.com/alwx/31d0d1b85d7ee26e8fbf11b696c09a96) and POSTed it without any problems:

```
~ http POST http://localhost:5005/model/test/intents\?model\=models/20210209-143218.tar.gz @data.json -v
POST /model/test/intents?model=models/20210209-143218.tar.gz HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 4354
Content-Type: application/json
Host: localhost:5005
User-Agent: HTTPie/2.3.0

{
    "rasa_nlu_data": {
       ...
    }
}


HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: filename
Connection: keep-alive
Content-Length: 7510
Content-Type: application/json
Keep-Alive: 5

{
    "entity_evaluation": {},
    "intent_evaluation": {
        "accuracy": 0.9855072464,
        ...
    },
    "response_selection_evaluation": null
}
```

So this PR updates the spec, adds more tests and fixes a small issue with the filename.
There is also a [Rasa X PR](https://github.com/RasaHQ/rasa-x/pull/4415) that changes the format of the data that's being sent from JSON to YAML on the Rasa X side just for the sake of consistency.

**Status (please check what you already did)**:
- [x] added some tests for the functionality
- [x] updated the spec
- [x] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/main/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
POST `/model/test/intents` endpoint fails
**Rasa version**: 2.0.0

**Python version**: 3.7

**Operating system** (windows, osx, ...): osx

**Issue**:
Calling POST `/model/test/intents` endpoint from `rasa x` with correct `model` path and body causes `500 Internal Server Error`


**Error (including full traceback)**:
```
{
    "version": "2.0.0",
    "status": "failure",
    "message": "An unexpected error occurred during evaluation. Error: Unknown data format for file '/var/folders/7y/mgv9hfvs45l75j9mn_5gxv2h0000gr/T/tmp93blnen6.md'.",
    "reason": "TestingError",
    "details": {},
    "help": null,
    "code": 500
}

2020-10-19 15:48:44 ERROR    rasa.server  - Traceback (most recent call last):
  File "/Users/rasa/.pyenv/versions/3.7.6/envs/rasa_376/lib/python3.7/site-packages/rasa/server.py", line 891, in evaluate_intents
    evaluation = run_evaluation(data_path, nlu_model, disable_plotting=True)
  File "/Users/rasa/.pyenv/versions/3.7.6/envs/rasa_376/lib/python3.7/site-packages/rasa/nlu/test.py", line 1476, in run_evaluation
    data_path, interpreter.model_metadata.language
  File "/Users/rasa/.pyenv/versions/3.7.6/envs/rasa_376/lib/python3.7/site-packages/rasa/shared/nlu/training_data/loading.py", line 60, in load_data
    data_sets = [_load(f, language) for f in files]
  File "/Users/rasa/.pyenv/versions/3.7.6/envs/rasa_376/lib/python3.7/site-packages/rasa/shared/nlu/training_data/loading.py", line 60, in <listcomp>
    data_sets = [_load(f, language) for f in files]
  File "/Users/rasa/.pyenv/versions/3.7.6/envs/rasa_376/lib/python3.7/site-packages/rasa/shared/nlu/training_data/loading.py", line 107, in _load
    raise ValueError(f"Unknown data format for file '{filename}'.")
ValueError: Unknown data format for file '/var/folders/7y/mgv9hfvs45l75j9mn_5gxv2h0000gr/T/tmp93blnen6.md'.

2020-10-19 15:48:44 ERROR    rasa.server  - An unexpected error occurred during evaluation. Error: Unknown data format for file '/var/folders/7y/mgv9hfvs45l75j9mn_5gxv2h0000gr/T/tmp93blnen6.md'.
```

**Command or request that led to error**:
```
POST localhost:5002/model/test/intents?model=20201014-170303.tar.gz
```

Hints from issue discussion:
my findings so far:

the request body I provided is in JSON format (which I get when calling from `rasa x`). It seems like the support for JSON was removed in 2.0
@federicotdn @degiz Can one of you please confirm? Do we need to update the API spec then?
I can definitely confirm that, since we've been debugging this issue together with @kalkbrennerei 

I'd rather fix this problem that just update the spec. 
@degiz @kalkbrennerei Could it be that the issue is that [this](https://github.com/RasaHQ/rasa/blob/aec8e57425dae174c401f2add3f7dd428f6c5977/rasa/server.py#L1150) stores the file as `.md` while we send a `json`? Rather seems like a Rasa X issue then as it should send yaml
@kalkbrennerei What's the state of this issue? can it be closed?
Ohhhh, I think the description is still up to date... But I also didn't work on it 😬 I just tried to run it again with rasa 2.2.7 and it's the same. It's not scale squad related so I will unassign...
I've just run into this also with 2.2. Calling Rasa OSS directly:

```sh
curl -X POST -d '{"model":"20201103-121624.tar.gz"}'  http://localhost:5005/model/test/intents?token=xxxxxxxxxxx
```

Returns:

```json
{"version":"2.2.7","status":"failure","message":"An unexpected error occurred during evaluation. Error: Unknown data format for file '\/tmp\/tmpvcrn8eod'.","reason":"TestingError","details":{},"help":null,"code":500}
```
This is a report from another system. In this case the return status is 200.

```sh
curl -X POST -d '{"model":"20210121-211252.tar.gz"}' http://localhost:5010/model/test/intents -v
* About to connect() to localhost port 5010 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5010 (#0)
> POST /model/test/intents HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:5010
> Accept: */*
> Content-Length: 34
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 34 out of 34 bytes
< HTTP/1.1 200 OK
< Access-Control-Expose-Headers: filename
< Access-Control-Allow-Credentials: true
< Content-Length: 88
< Content-Type: application/json
< Connection: keep-alive
< Keep-Alive: 5
< 
* Connection #0 to host localhost left intact
{"intent_evaluation":null,"entity_evaluation":null,"response_selection_evaluation":null}
```
@rgstephens could you please share more details about the project and model that was used in the first case? Did it contain any `.md` files?  I tried to reproduce it with a new project, and got this:

```
(base) ➜  ~ curl -X POST -d '{"model":"20210208-160756.tar.gz"}' http://localhost:5005/model/test/intents -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5005 (#0)
> POST /model/test/intents HTTP/1.1
> Host: localhost:5005
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 34
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 34 out of 34 bytes
< HTTP/1.1 200 OK
< Access-Control-Expose-Headers: filename
< Access-Control-Allow-Credentials: true
< Content-Length: 88
< Content-Type: application/json
< Connection: keep-alive
< Keep-Alive: 5
<
* Connection #0 to host localhost left intact
{"intent_evaluation":null,"entity_evaluation":null,"response_selection_evaluation":null}
```
@alwx 

Here are the details for the model @rgstephens mentioned. This was trained from the the helpdesk assistant example, but It's also happening with models we've trained from the ground up.

![rasa_model_structure](https://user-images.githubusercontent.com/78817599/107409632-36780600-6ac1-11eb-9a4b-eacc5fdc685a.png)

Commit message: Updated schema and md changelog entry

Patch to review:
diff --git a/changelog/7063.bugfix.md b/changelog/7063.bugfix.md
new file mode 100644
index 000000000000..660c79a84b83
--- /dev/null
+++ b/changelog/7063.bugfix.md
@@ -0,0 +1,4 @@
+Update the spec of `POST /model/test/intents` and add tests for cases when JSON is provided.
+
+Fix the incorrect temporary file extension for the data that gets extracted from the payload provided
+in the body of `POST /model/test/intents` request.
\ No newline at end of file
diff --git a/docs/static/spec/rasa.yml b/docs/static/spec/rasa.yml
index 7d69a6154e1a..dd8ce059f56e 100644
--- a/docs/static/spec/rasa.yml
+++ b/docs/static/spec/rasa.yml
@@ -566,6 +566,12 @@ paths:
       requestBody:
         required: true
         content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                rasa_nlu_data:
+                  $ref: '#/components/schemas/RasaNLUData'
           application/x-yaml:
             schema:
               type: string
@@ -1802,3 +1808,29 @@ components:
         properties:
           use_entities:
             type: boolean
+
+    RasaNLUData:
+      type: object
+      properties:
+        common_examples:
+          type: object
+          items:
+            type: array
+            items:
+              $ref: '#/components/schemas/CommonExample'
+
+    CommonExample:
+      type: object
+      properties:
+        entities:
+          description: Expected entities
+          type: array
+          items:
+            $ref: '#/components/schemas/Entity'
+        intent:
+          type: string
+          description: Intent name
+        text:
+          type: string
+          description: Text of the message
+          example: "Hello!"
diff --git a/rasa/server.py b/rasa/server.py
index 4ec08ac518f9..9649721afedf 100644
--- a/rasa/server.py
+++ b/rasa/server.py
@@ -1066,7 +1066,7 @@ async def evaluate_stories(
             "evaluate your model.",
         )
 
-        test_data = _test_data_file_from_payload(request, temporary_directory)
+        test_data = _test_data_file_from_payload(request, temporary_directory, ".md")
 
         use_e2e = rasa.utils.endpoints.bool_arg(request, "e2e", default=False)
 
@@ -1397,14 +1397,16 @@ def _get_output_channel(
     )
 
 
-def _test_data_file_from_payload(request: Request, temporary_directory: Path) -> Text:
+def _test_data_file_from_payload(
+    request: Request, temporary_directory: Path, suffix: Text = ".tmp"
+) -> Text:
     if request.headers.get("Content-type") == YAML_CONTENT_TYPE:
         return str(
             _training_payload_from_yaml(request, temporary_directory)["training_files"]
         )
     else:
         return rasa.utils.io.create_temporary_file(
-            request.body, mode="w+b", suffix=".md"
+            request.body, mode="w+b", suffix=suffix
         )
 
 
diff --git a/tests/test_server.py b/tests/test_server.py
index e1a79cba870f..a66f73a78010 100644
--- a/tests/test_server.py
+++ b/tests/test_server.py
@@ -754,7 +754,7 @@ async def test_evaluate_stories_end_to_end(
 ):
     stories = rasa.shared.utils.io.read_file(end_to_end_test_story_file)
 
-    _, response = await rasa_app.post("/model/test/stories?e2e=true", data=stories)
+    _, response = await rasa_app.post("/model/test/stories?e2e=true", data=stories,)
 
     assert response.status == HTTPStatus.OK
     js = response.json()
@@ -794,6 +794,47 @@ async def test_evaluate_intent(rasa_app: SanicASGITestClient, default_nlu_data:
     }
 
 
+async def test_evaluate_intent_json(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
+):
+    nlu_data = rasa.shared.utils.io.read_file("data/test/demo-rasa-small.json")
+
+    _, response = await rasa_app.post(
+        "/model/test/intents",
+        json=nlu_data,
+        headers={"Content-type": rasa.server.JSON_CONTENT_TYPE},
+    )
+
+    assert response.status == HTTPStatus.OK
+    assert set(response.json().keys()) == {
+        "intent_evaluation",
+        "entity_evaluation",
+        "response_selection_evaluation",
+    }
+
+
+async def test_evaluate_invalid_intent_model_file(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
+):
+    _, response = await rasa_app.post(
+        "/model/test/intents?model=invalid.tar.gz",
+        json={},
+        headers={"Content-type": rasa.server.JSON_CONTENT_TYPE},
+    )
+
+    assert response.status == HTTPStatus.INTERNAL_SERVER_ERROR
+
+
+async def test_evaluate_intent_without_body(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
+):
+    _, response = await rasa_app.post(
+        "/model/test/intents", headers={"Content-type": rasa.server.YAML_CONTENT_TYPE},
+    )
+
+    assert response.status == HTTPStatus.BAD_REQUEST
+
+
 async def test_evaluate_intent_on_just_nlu_model(
     rasa_app_nlu: SanicASGITestClient, default_nlu_data: Text
 ):


Code review comments:
diff_hunk: @@ -794,6 +794,47 @@ async def test_evaluate_intent(rasa_app: SanicASGITestClient, default_nlu_data:
     }
 
 
+async def test_evaluate_intent_json(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
path: tests/test_server.py
text: @user1:
```suggestion
    rasa_app: SanicASGITestClient
```

diff_hunk: @@ -794,6 +794,47 @@ async def test_evaluate_intent(rasa_app: SanicASGITestClient, default_nlu_data:
     }
 
 
+async def test_evaluate_intent_json(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
+):
+    nlu_data = rasa.shared.utils.io.read_file("data/test/demo-rasa-small.json")
+
+    _, response = await rasa_app.post(
+        "/model/test/intents",
+        json=nlu_data,
+        headers={"Content-type": rasa.server.JSON_CONTENT_TYPE},
+    )
+
+    assert response.status == HTTPStatus.OK
+    assert set(response.json().keys()) == {
+        "intent_evaluation",
+        "entity_evaluation",
+        "response_selection_evaluation",
+    }
+
+
+async def test_evaluate_invalid_intent_model_file(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
+):
+    _, response = await rasa_app.post(
+        "/model/test/intents?model=invalid.tar.gz",
+        json={},
+        headers={"Content-type": rasa.server.JSON_CONTENT_TYPE},
+    )
+
+    assert response.status == HTTPStatus.INTERNAL_SERVER_ERROR
+
+
+async def test_evaluate_intent_without_body(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
path: tests/test_server.py
text: @user1:
```suggestion
    rasa_app: SanicASGITestClient
```

diff_hunk: @@ -794,6 +794,47 @@ async def test_evaluate_intent(rasa_app: SanicASGITestClient, default_nlu_data:
     }
 
 
+async def test_evaluate_intent_json(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
+):
+    nlu_data = rasa.shared.utils.io.read_file("data/test/demo-rasa-small.json")
+
+    _, response = await rasa_app.post(
+        "/model/test/intents",
+        json=nlu_data,
+        headers={"Content-type": rasa.server.JSON_CONTENT_TYPE},
+    )
+
+    assert response.status == HTTPStatus.OK
+    assert set(response.json().keys()) == {
+        "intent_evaluation",
+        "entity_evaluation",
+        "response_selection_evaluation",
+    }
+
+
+async def test_evaluate_invalid_intent_model_file(
+    rasa_app: SanicASGITestClient, default_nlu_data: Text
path: tests/test_server.py
text: @user1:
```suggestion
    rasa_app: SanicASGITestClient,
```