Repository: OctoPrint/OctoPrint

Issue:
additional_state_data is broken with invalid JSON
<!--
READ THE FOLLOWING FIRST:

If not already done, please read the the Contribution Guidelines that
are linked to the right under "Helpful resources" > "Contributing".

Also read the FAQ: https://faq.octoprint.org

This is a bug and feature tracker, please only use it to report bugs
or request features within OctoPrint (not OctoPi, not any OctoPrint
plugins and not unofficial OctoPrint versions).

Do not seek support here ("I need help with ...", "I have a
question ..."), that belongs on the community forum at
community.octoprint.org, NOT here.

Mark requests with a "[Request]" prefix in the title please. For bug
reports fully fill out the bug reporting template (if you don't know
where to find some information - it's all described in the Contribution
Guidelines linked up there in the big yellow box).

When reporting a bug do NOT delete ANY lines from the template.

Make sure any bug you want to report is still present with the CURRENT
OctoPrint version and that it does not vanish when you start OctoPrint
in safe mode - how to do that is also explained in the Contribution
Guidelines linked up there in the big yellow box.

Thank you!
-->

#### What were you doing?

Trying to use the additional_state_data hook to send a dict with a value that is `float("inf")`.

1. Create a plugin in the normal way.
2. Add to the hooks: `"octoprint.printer.additional_state_data": __plugin_implementation__.additional_state_data`
3. Add this function:

```python
  def additional_state_data(self, initial, *args, **kwargs):
    return {"x": float("inf")}
```
#### What did you expect to happen?

I expected that the print time left would still be displayed on screen.

#### What happened instead?

Print time left is no longer displayed on screen.  Printer is crippled, unable to print from GUI.

#### Did the same happen when running OctoPrint in safe mode?

I didn't try this because this bug can only happen if plugins are enabled.

#### Version of OctoPrint

OctoPrint 1.5.1 Python 3.7.7

#### Operating System running OctoPrint

Linux

#### Printer model & used firmware incl. version

Virtual printer

#### Browser and version of browser, operating system running browser

Chrome

#### Link to contents of terminal tab or serial.log

#### Link to contents of Javascript console in the browser

#### Screenshot(s)/video(s) showing the problem:

I have read the FAQ.
additional_state_data is broken with invalid JSON
<!--
READ THE FOLLOWING FIRST:

If not already done, please read the the Contribution Guidelines that
are linked to the right under "Helpful resources" > "Contributing".

Also read the FAQ: https://faq.octoprint.org

This is a bug and feature tracker, please only use it to report bugs
or request features within OctoPrint (not OctoPi, not any OctoPrint
plugins and not unofficial OctoPrint versions).

Do not seek support here ("I need help with ...", "I have a
question ..."), that belongs on the community forum at
community.octoprint.org, NOT here.

Mark requests with a "[Request]" prefix in the title please. For bug
reports fully fill out the bug reporting template (if you don't know
where to find some information - it's all described in the Contribution
Guidelines linked up there in the big yellow box).

When reporting a bug do NOT delete ANY lines from the template.

Make sure any bug you want to report is still present with the CURRENT
OctoPrint version and that it does not vanish when you start OctoPrint
in safe mode - how to do that is also explained in the Contribution
Guidelines linked up there in the big yellow box.

Thank you!
-->

#### What were you doing?

Trying to use the additional_state_data hook to send a dict with a value that is `float("inf")`.

1. Create a plugin in the normal way.
2. Add to the hooks: `"octoprint.printer.additional_state_data": __plugin_implementation__.additional_state_data`
3. Add this function:

```python
  def additional_state_data(self, initial, *args, **kwargs):
    return {"x": float("inf")}
```
#### What did you expect to happen?

I expected that the print time left would still be displayed on screen.

#### What happened instead?

Print time left is no longer displayed on screen.  Printer is crippled, unable to print from GUI.

#### Did the same happen when running OctoPrint in safe mode?

I didn't try this because this bug can only happen if plugins are enabled.

#### Version of OctoPrint

OctoPrint 1.5.1 Python 3.7.7

#### Operating System running OctoPrint

Linux

#### Printer model & used firmware incl. version

Virtual printer

#### Browser and version of browser, operating system running browser

Chrome

#### Link to contents of terminal tab or serial.log

#### Link to contents of Javascript console in the browser

#### Screenshot(s)/video(s) showing the problem:

I have read the FAQ.


Discussion:
No logs here? Looks like that part of the template was deleted or ignored 🤔 

I haven't got time to reproduce this at the moment & find the error myself. A log would be great, to save myself time in figuring out how to break something you've already done. It will get looked at by someone, but sooner if there are logs.
I didn't bother with the log because there's nothing of interest in it.  The bug is that OctoPrint thinks that it is sending valid JSON but it is not.  The web UI is trying to parse that invalid UI and failing.  The web UI doesn't show any error, though, which is a shame because it ought to have parsed the JSON instead of just consuming it.

I will do some more debugging on it.  I think that the solution is that OctoPrint sanitize the JSON before sending it.
Ok, the way the bug report was written sounded like there was obvious errors or something. We prefer logs to be included even if you don't think they are relevant, because of the whole system/environment, what events have happened. With exceptions, people start to not provide logs & waste our time.

I understand the problem a *lot* better now with that follow up comment, thanks. Maybe a simple `flask.jsonify` or something should be used before the request goes out.
I would like to make it easier for you to debug.  If I sent you a link to a plugin that you can install, would that help?

https://github.com/eyal0/OctoPrint-TimeToFilament/archive/break_json.zip

Install that into your Octoprint.  It ought to make your printer completely not functional.

In your browser (I used Chrome), try hitting "F12" to open up the developer console.  Then go to the "Network" tab and look for the "websocket" transfer, which should be on-going and it never ends.  Double click on it.  Then select the "Messages" tab and you'll see about 2 messages per second.  Each one is *supposed* to be JSON but they aren't and that breaks the webUI.

Here's what it looks like when it works:

![image](https://user-images.githubusercontent.com/109809/101842319-ff0ecc80-3b04-11eb-918b-3a11d3ebe19a.png)

Notice "Network" is selected at the top.  Notice that "websocket" is selected on the left panel.  Notice "Messages" is selected for the websocket.  Notice that I selected a message.  Finally, notice that the contents of the message were recognized as JSON and so the bottom right shows a tree view of the data, with element `a` and sub-element `0`.

Now again but with the broken plugin:

![image](https://user-images.githubusercontent.com/109809/101842535-7cd2d800-3b05-11eb-87cb-025f42b179f9.png)

Same as before but the text in the bottom right is shown as text instead of a tree view because Chrome did not recognize the text as valid JSON.  The text is:

```
a[{"current":{"state":{"text":"Operational","flags":{"operational":true,"printing":false,"cancelling":false,"pausing":false,"resuming":false,"finishing":false,"closedOrError":false,"error":false,"paused":false,"ready":true,"sdReady":true}},"job":{"file":{"name":null,"path":null,"size":null,"origin":null,"date":null},"estimatedPrintTime":null,"lastPrintTime":null,"filament":{"length":null,"volume":null},"user":null},"currentZ":null,"progress":{"completion":null,"filepos":null,"printTime":null,"printTimeLeft":null,"printTimeOrigin":null},"offsets":{},"resends":{"count":0,"transmitted":8,"ratio":0},"plugins":{"TimeToFilament":{"x":Infinity}},"serverTime":1607643217.940848,"temps":[{"time":1607643217,"tool0":{"actual":21.3,"target":0.0},"bed":{"actual":21.3,"target":0.0},"chamber":{"actual":null,"target":null}}],"logs":["Recv: T:21.30/ 0.00 B:21.30/ 0.00 @:64","Recv: Not SD printing"],"messages":["T:21.30/ 0.00 B:21.30/ 0.00 @:64","Not SD printing"],"busyFiles":[]}}]
```

If I remove the leading `a[` and trailing `]` then I can paste it into an online JSON parser like this one: http://json.parser.online.fr/

It parses what it can but also reports an error:

`SyntaxError: Unexpected token I in JSON at position 634`

This corresponds to the bare word `Infinity` in the text, which is not valid JSON.

What's strange is that the python string for infinity is not `Infinity`, it's `inf`.  So it seems like the python side of the `self._emit` function is trying to create something that javascript can read.  It's not strict JSON but `Infinity` is a valid identifier in javascript.  So a JSON parser on the javascript side that isn't too strict could potentially figure out what `Infinity` means and create the right object.  It would be non-standard JSON but it would work!

Barring that, the python should make sure that what it gets from the `additional_state_data` hook is strictly JSON.  Maybe convert from dict to JSON and then back to dict and report errors if there are any.  That would solve the problem, I think, though it wouldn't allow for using "Infinity" like I wanted to.

I hope that helps.

I have searched the sockjs code, both sockjs client and tornado sockjs server, and I can't find any mention of "Infinity".  I'm not sure how it's happening but somewhere, the message coming from the server that has a python infinity in it, which is `inf`, is getting converted to `Infinity`.  I don't see where it is happening!
Alright, looks like that is built-in to the python json parser.

```
$ python -c 'import json; print(json.dumps({"x": float("inf")}))'
{"x": Infinity}
```

So that's where the "Infinity" is coming from.  The JSON parser on the javascript side is shitting the bed when it gets this.
One solution would be to make the parser on the receiving side more forgiving.  Here are some solutions: https://stackoverflow.com/questions/21896792/force-json-stringify-to-emit-nan-infinity-or-js-json-lib-that-does-so

Another possibility is to make the JSON encoding on the sending side more strict.  There is an option to the python json dumps funtion called `allow_nan` that will do it.

I think that might be best.  Modifying the parsing of JSON on the javascript side would be a pain in the ass.  So let's just make it report an error in python!  Easy.  I will make a PR.
No logs here? Looks like that part of the template was deleted or ignored 🤔 

I haven't got time to reproduce this at the moment & find the error myself. A log would be great, to save myself time in figuring out how to break something you've already done. It will get looked at by someone, but sooner if there are logs.
I didn't bother with the log because there's nothing of interest in it.  The bug is that OctoPrint thinks that it is sending valid JSON but it is not.  The web UI is trying to parse that invalid UI and failing.  The web UI doesn't show any error, though, which is a shame because it ought to have parsed the JSON instead of just consuming it.

I will do some more debugging on it.  I think that the solution is that OctoPrint sanitize the JSON before sending it.
Ok, the way the bug report was written sounded like there was obvious errors or something. We prefer logs to be included even if you don't think they are relevant, because of the whole system/environment, what events have happened. With exceptions, people start to not provide logs & waste our time.

I understand the problem a *lot* better now with that follow up comment, thanks. Maybe a simple `flask.jsonify` or something should be used before the request goes out.
I would like to make it easier for you to debug.  If I sent you a link to a plugin that you can install, would that help?

https://github.com/eyal0/OctoPrint-TimeToFilament/archive/break_json.zip

Install that into your Octoprint.  It ought to make your printer completely not functional.

In your browser (I used Chrome), try hitting "F12" to open up the developer console.  Then go to the "Network" tab and look for the "websocket" transfer, which should be on-going and it never ends.  Double click on it.  Then select the "Messages" tab and you'll see about 2 messages per second.  Each one is *supposed* to be JSON but they aren't and that breaks the webUI.

Here's what it looks like when it works:

![image](https://user-images.githubusercontent.com/109809/101842319-ff0ecc80-3b04-11eb-918b-3a11d3ebe19a.png)

Notice "Network" is selected at the top.  Notice that "websocket" is selected on the left panel.  Notice "Messages" is selected for the websocket.  Notice that I selected a message.  Finally, notice that the contents of the message were recognized as JSON and so the bottom right shows a tree view of the data, with element `a` and sub-element `0`.

Now again but with the broken plugin:

![image](https://user-images.githubusercontent.com/109809/101842535-7cd2d800-3b05-11eb-87cb-025f42b179f9.png)

Same as before but the text in the bottom right is shown as text instead of a tree view because Chrome did not recognize the text as valid JSON.  The text is:

```
a[{"current":{"state":{"text":"Operational","flags":{"operational":true,"printing":false,"cancelling":false,"pausing":false,"resuming":false,"finishing":false,"closedOrError":false,"error":false,"paused":false,"ready":true,"sdReady":true}},"job":{"file":{"name":null,"path":null,"size":null,"origin":null,"date":null},"estimatedPrintTime":null,"lastPrintTime":null,"filament":{"length":null,"volume":null},"user":null},"currentZ":null,"progress":{"completion":null,"filepos":null,"printTime":null,"printTimeLeft":null,"printTimeOrigin":null},"offsets":{},"resends":{"count":0,"transmitted":8,"ratio":0},"plugins":{"TimeToFilament":{"x":Infinity}},"serverTime":1607643217.940848,"temps":[{"time":1607643217,"tool0":{"actual":21.3,"target":0.0},"bed":{"actual":21.3,"target":0.0},"chamber":{"actual":null,"target":null}}],"logs":["Recv: T:21.30/ 0.00 B:21.30/ 0.00 @:64","Recv: Not SD printing"],"messages":["T:21.30/ 0.00 B:21.30/ 0.00 @:64","Not SD printing"],"busyFiles":[]}}]
```

If I remove the leading `a[` and trailing `]` then I can paste it into an online JSON parser like this one: http://json.parser.online.fr/

It parses what it can but also reports an error:

`SyntaxError: Unexpected token I in JSON at position 634`

This corresponds to the bare word `Infinity` in the text, which is not valid JSON.

What's strange is that the python string for infinity is not `Infinity`, it's `inf`.  So it seems like the python side of the `self._emit` function is trying to create something that javascript can read.  It's not strict JSON but `Infinity` is a valid identifier in javascript.  So a JSON parser on the javascript side that isn't too strict could potentially figure out what `Infinity` means and create the right object.  It would be non-standard JSON but it would work!

Barring that, the python should make sure that what it gets from the `additional_state_data` hook is strictly JSON.  Maybe convert from dict to JSON and then back to dict and report errors if there are any.  That would solve the problem, I think, though it wouldn't allow for using "Infinity" like I wanted to.

I hope that helps.

I have searched the sockjs code, both sockjs client and tornado sockjs server, and I can't find any mention of "Infinity".  I'm not sure how it's happening but somewhere, the message coming from the server that has a python infinity in it, which is `inf`, is getting converted to `Infinity`.  I don't see where it is happening!
Alright, looks like that is built-in to the python json parser.

```
$ python -c 'import json; print(json.dumps({"x": float("inf")}))'
{"x": Infinity}
```

So that's where the "Infinity" is coming from.  The JSON parser on the javascript side is shitting the bed when it gets this.
One solution would be to make the parser on the receiving side more forgiving.  Here are some solutions: https://stackoverflow.com/questions/21896792/force-json-stringify-to-emit-nan-infinity-or-js-json-lib-that-does-so

Another possibility is to make the JSON encoding on the sending side more strict.  There is an option to the python json dumps funtion called `allow_nan` that will do it.

I think that might be best.  Modifying the parsing of JSON on the javascript side would be a pain in the ass.  So let's just make it report an error in python!  Easy.  I will make a PR.

Resolving patch:
diff --git a/src/octoprint/printer/standard.py b/src/octoprint/printer/standard.py
--- a/src/octoprint/printer/standard.py
+++ b/src/octoprint/printer/standard.py
@@ -18,6 +18,7 @@
 from frozendict import frozendict
 from past.builtins import basestring, long
 
+import octoprint.util.json
 from octoprint import util as util
 from octoprint.events import Events, eventManager
 from octoprint.filemanager import FileDestinations, NoSuchStorage, valid_file_type
@@ -272,7 +273,13 @@ def _get_additional_plugin_data(self, initial=False):
             try:
                 additional = hook(initial=initial)
                 if additional and isinstance(additional, dict):
+                    octoprint.util.json.dump({name: additional})
                     plugin_data[name] = additional
+            except ValueError:
+                self._logger.exception(
+                    "Invalid additional data from plugin {}".format(name),
+                    extra={"plugin": name},
+                )
             except Exception:
                 self._logger.exception(
                     "Error while retrieving additional data from plugin {}, blacklisting it for further loops".format(
diff --git a/src/octoprint/server/util/sockjs.py b/src/octoprint/server/util/sockjs.py
--- a/src/octoprint/server/util/sockjs.py
+++ b/src/octoprint/server/util/sockjs.py
@@ -5,7 +5,6 @@
 __license__ = "GNU Affero General Public License http://www.gnu.org/licenses/agpl.html"
 __copyright__ = "Copyright (C) 2014 The OctoPrint Project - Released under terms of the AGPLv3 License"
 
-import json
 import logging
 import threading
 import time
@@ -27,7 +26,7 @@
 from octoprint.access.users import LoginStatusListener
 from octoprint.events import Events
 from octoprint.settings import settings
-from octoprint.util.json import JsonEncoding
+from octoprint.util.json import dump as json_dump
 
 
 class ThreadSafeSession(octoprint.vendor.sockjs.tornado.session.Session):
@@ -67,11 +66,7 @@ def remove_handler(self, handler):
 class JsonEncodingSessionWrapper(wrapt.ObjectProxy):
     def send_message(self, msg, stats=True, binary=False):
         self.send_jsonified(
-            json.dumps(
-                octoprint.vendor.sockjs.tornado.util.bytes_to_str(msg),
-                separators=(",", ":"),
-                default=JsonEncoding.encode,
-            ),
+            json_dump(octoprint.vendor.sockjs.tornado.util.bytes_to_str(msg)),
             stats,
         )
 
diff --git a/src/octoprint/util/json/__init__.py b/src/octoprint/util/json/__init__.py
--- a/src/octoprint/util/json/__init__.py
+++ b/src/octoprint/util/json/__init__.py
@@ -6,12 +6,22 @@
 
 
 import collections
+import json
 
 import frozendict
 
 from octoprint.util import to_unicode
 
 
+def dump(obj):
+    return json.dumps(
+        obj,
+        separators=(",", ":"),
+        default=JsonEncoding.encode,
+        allow_nan=False,
+    )
+
+
 class JsonEncoding(object):
 
     encoders = collections.OrderedDict()


Test patch: