Repository: pyodide/pyodide

Issue:
Passing python dict into a js function
Hi, I just noticed that we now (v0.17.0) convert python dict to a `Map` in js, and it break my previous code.

I have 3 questions:
1. What would be the easiest way to send a dictionary in Python to Js so it will become a regular object. 
2. What is the reason to choose converting `dict` to `Map`? I find it's bit inconvenient if we get a Map instead of an object by default. I mean, `Object`/`{}` is obviously more commonly used than `Map` in JS, wouldn't it be a better default type for converting `dict` in Python?
3. Previously, if we import js function (e.g. `postMessage`) into Python, then send a python dictionary( `js.postMessage({'data': 123})`, it automatically convert the arguments into js object, but now it complains that `JsException: DataCloneError: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': [object Object] could not be cloned.`. 





Discussion:
Hi @oeway, thanks for the question. We didn't have time to vet this as well as I would have liked and so there are some usability issues. Ideally it would be good to have a careful discussion about each detail like this, but since we have such limited developer time available on this project, there's inevitably going to be things that we don't consider carefully enough.

Though also this could have been discussed before the release if we'd made a beta release and given people a week or two with it. The four days we allowed for that was obviously not enough.


3. You can do `js.postMessage(**{'data': 123})` or `js.postMessage(data=123)`. Also, it's okay to do `from js import postMessage` and use `postMessage` directly now if you prefer.

.

1. If the object is the last argument:
```py
jsfunc(arg1, arg2, **my_dict)
```
If the object is not the last argument:
```py
from js import Object
from pyodide import to_js
jsfunc(Object.from_entries(to_js(my_dict)), arg2)
```

2. There are several reasons pointing in each direction. The main reason I chose `Map` is that Objects only can have strings as keys, whereas `Map` can have anything. (This isn't as beneficial as I thought though, because only strings or numbers are looked up by value, everything else keys on object identity.) Going from only Strings to Strings or Numbers isn't necessarily enough of a benefit to make it worth the annoyance.

Also, on a PyProxy of a dict you would look up a key with `proxy.get(key)`. If we convert it to a map, then `proxy.toJs().get(key)` does the same thing. It seemed nice to keep the `getattr`/`getitem` distinction if possible.

Also, `pyodide.toPy(my_dict.toJs())` will give back a dict equal to the original one this way, whereas with an object, `{ 1 : 2}` will turn into `{"1" : 2}`.

But at the moment I am thinking it might have been a better idea to keep the `Object`.

Ideally we should make it so that `toJs` gives people some choice about how things are converted.
Hi @hoodmane Thanks for you prompt answer! Sure I understand that, and the team have been making tremendous progress!

Regarding 1 and 3, if I have nested dictionaries, that won't work, right? Would it make sense to restore the behaviour that automatically convert nested dictionaries automatically into nested objects when calling js functions? 

For 2, I think it might worth more discussion on usability vs accuracy here. What you described all make sense, but it does reduce the usability. 

Let me also listed a few cases where Object is more practical rather than Map:

For the following python code:
```python
map = {"key1": "value1", "key2": "value2"}
print(map["key1"])
```

Using Map would be like this:
```js
let map = new Map([["key1", "value1"], ["key2", "value2"]])
console.log(map.get("key1"))
```

But if we use object in js, the same python code can essentially work with little change:
```js
let map = {"key1": "value1", "key2": "value2"}
console.log(map["key1"])
```

Another thing is that, if we do JSON serialisation on Map, it won't work directly:
```js
let map = new Map([["key1", "value1"], ["key2", "value2"]])
JSON.stringify(map) // returns "{}"
```

> Also, pyodide.toPy(my_dict.toJs()) will give back a dict equal to the original one this way, whereas with an object, { 1 : 2} will turn into {"1" : 2}.

I think it's not exactly {"1": 2}, because in JS you can also use number as index for object:
```js
let a=1, b={}
b[a] = 2

b[1] === 2 // true
b["1"] === 2 // true
```
This will then alleviate the conversion issue? This means, it only becomes an issue if we have both `1` and `"1"` as key in a python dictionary and we convert to JS object they cannot co-exist.  However, I would say, this is very rare in practice, most people will just use string as key and if they want to use number they will not mix with string keys, or they might just go for list/Array -- it will be very confusing if they start to mix number-string and numbers as key in the same dictionary.

What do you think?



> I have nested dictionaries, that won't work, right?

Right.

> it might worth more discussion on usability vs accuracy here.

Underlying problem is that I am a mathematician and sometimes I suffer from obsessive accuracy. It's important to get feedback to check that occasionally.

> Another thing is that, if we do JSON serialisation on Map, it won't work directly:

=(

> it will be very confusing if they start to mix number-string and numbers as key in the same dictionary.

I agree that this use case is quite rare. In particular, the vast majority of the time dict keys will have only one type. In the rare case that keys are mixed types, they will very very likely be mixed between a bunch of different types and we can only handle strings, numbers, or one extra type.

One extra issue with `Object` is what if the dict has keys that overlap with `Object.prototype`? Options:
1. Throw error
2. Use `Object.create(null)`.

What do you think about `Object.create(null)`?
Anyways you have convinced me that this needs to be changed. In fact, I was 90% convinced when I realized that `Object.toJs` wouldn't work directly with `fetch`:
https://pyodide.org/en/stable/usage/faq.html#how-can-i-use-fetch-with-optional-arguments-from-python

This should have been fixed in a beta release though. As I said above, next time we need to make a beta release and ensure that people have a more reasonable amount of time with it.
>  This means, it only becomes an issue if we have both 1 and "1" as key in a python dictionary and we convert to JS object they cannot co-exist.

Wasn't the issue also that if you have a `Dict[int, Any]` if you do a round trip conversion to JS you would end up with `Dict[str, Any]` ? Or I guess that only would be an issue with an explicit conversion.
> Anyways you have convinced me that this needs to be changed

We could consider it a bug fix for 0.17.1 if necessary, but it would be good to get more user feedback from existing users as well.
> if you have a `Dict[int, Any]` if you do a round trip conversion to JS you would end up with `Dict[str, Any]`

Yeah that was the issue. This is alleviated by proxying dicts, but even with explicit conversions it's still super undesirable.

> a bug fix for 0.17.1

So the options are:

* Keep current behavior but add variations or more flexibility to toJs to make object case more ergonomic
* Change default to `Object`, add more flexibility to allow `Map` when desired.
* Use `Object.create(null)` (cleaner than `Object` because there are no `Object.prototype` name clashes but some some things are not expecting it and break weirdly).

If we want feedback on this, maybe we should set up a thread about this with a poll, then tweet it out?
Sounds good. Maybe we could indeed open a RFC issue with these 3 options, examples and advantages/disadvantages of each, then advertise it a bit? A poll could be a good idea, but my concern is that it's also easy to vote without understanding all the technical implications, and this has a number of those.
> open a RFC issue

Sounds good. Make also make a way for people to subscribe to RFCs, then we can tweet / @ a bunch of people suggesting they might want to subscribe. The approach I used in #1331 certainly works for an RFC in a pinch, but it'd be good to give people the option to opt into the RFC rather than having me just look through threads for people who seem like they'd want to know.
Having the same issues as posted by @oeway. Been reviving a project which used 0.15.0, and upgrading to 0.17.0 broke it for the same reasons mentioned above. I'm keen to help with contribution if help is needed, but I might need a bit of hand holding initially, as my raw C experience is limited mainly to past university projects.
@ibdafna Well we still haven't quite decided what should be done. I'm wondering what an API that allows a choice of this behavior would look like. Maybe we could just take a flag `toJs({ dict_to_object : true })`? Anyways I think our plan is still to try to get more input on this.

If you have an idea of what the logic you would like is, it'd be great if you would open a PR but just be aware that we may not be able to merge if we decide on a different design.

I would be happy to answer any questions you have about how the code works.

The relevant function is here:
https://github.com/pyodide/pyodide/blob/2a906bd48969ca373d8421949b7b10b1f1ccb43a/src/core/python2js.c#L141
The functions `JsMap_New` and `JsMap_Set` are here:
https://github.com/pyodide/pyodide/blob/2a906bd48969ca373d8421949b7b10b1f1ccb43a/src/core/hiwire.c#L746

It's possible we could revert the behavior of this as 0.17.1 and possibly make a more flexible API in the future.
I bumped into this recently, and it was annoying, but it's not too hard to convert a nested map into an object.

```js
const toObject = (x) => {
  if (x instanceof Map) {
    return Object.fromEntries(Array.from(
      x.entries(),
      ([k, v]) => [k, toObject(v)]
    ))
  } else if (x instanceof Array) {
    return x.map(toObject);
  } else {
    return x;
  }
}
```
And from Python you could do e.g.,
```py
from js import fetch, toObject # if toObject is a global
from pyodide import to_js
fetch(url, toObject(to_js(options_dict)))
```

I think this is okay for now and we can plan to make a more convenient solution in v0.18.
I would prefer `fetch(url, options_dict)` ;) 
Looking forward to v0.18!
Unfortunately we can't support `fetch(url, options_dict)`.
Would love to see dict -> object conversion being added in the next release!



> If the object is not the last argument:
> 
> ```python
> from js import Object
> from pyodide import to_js
> jsfunc(Object.from_entries(to_js(my_dict)), arg2)
> ```
> 


I was a bit confused since I couldn't make this code work, turns out the correct call should be:
```python
jsfunc(Object.fromEntries(to_js(my_dict)), arg2)
```

Resolving patch:
diff --git a/src/py/_pyodide/_core_docs.py b/src/py/_pyodide/_core_docs.py
--- a/src/py/_pyodide/_core_docs.py
+++ b/src/py/_pyodide/_core_docs.py
@@ -1,6 +1,6 @@
 # type: ignore
 
-from typing import Any, Callable
+from typing import Any, Callable, Iterable
 
 # All docstrings for public `core` APIs should be extracted from here. We use
 # the utilities in `docstring.py` and `docstring.c` to format them
@@ -130,7 +130,8 @@ def to_js(
         *,
         depth: int = -1,
         pyproxies: JsProxy = None,
-        create_pyproxies: bool = True
+        create_pyproxies: bool = True,
+        dict_converter: Callable[[Iterable[JsProxy]], JsProxy] = None,
     ) -> JsProxy:
         """Convert the object to Javascript.
 
@@ -160,6 +161,15 @@ def to_js(
         create_pyproxies: bool, default=True
             If you set this to False, :any:`to_js` will raise an error
 
+        dict_converter: Callable[[Iterable[JsProxy]], JsProxy], defauilt = None
+            This converter if provided recieves a (Javascript) iterable of
+            (Javascript) pairs [key, value]. It is expected to return the
+            desired result of the dict conversion. Some suggested values for
+            this argument:
+
+                js.Map.new -- similar to the default behavior
+                js.Array.from -- convert to an array of entries
+                js.Object.fromEntries -- convert to a Javascript object
         """
         return obj
 


Test patch:
diff --git a/src/tests/test_typeconversions.py b/src/tests/test_typeconversions.py
--- a/src/tests/test_typeconversions.py
+++ b/src/tests/test_typeconversions.py
@@ -403,6 +403,63 @@ class Test: pass
     )
 
 
+def test_dict_converter(selenium):
+    assert (
+        selenium.run_js(
+            """
+            self.arrayFrom = Array.from;
+            return pyodide.runPython(`
+                from js import arrayFrom
+                from pyodide import to_js
+                res = to_js({ x : x + 2 for x in range(5)}, dict_converter=arrayFrom)
+                res
+            `)
+            """
+        )
+        == [[0, 2], [1, 3], [2, 4], [3, 5], [4, 6]]
+    )
+
+    assert (
+        selenium.run_js(
+            """
+            let px = pyodide.runPython("{ x : x + 2 for x in range(5)}");
+            let result = px.toJs({dict_converter : Array.from});
+            px.destroy();
+            return result;
+            """
+        )
+        == [[0, 2], [1, 3], [2, 4], [3, 5], [4, 6]]
+    )
+
+    assert (
+        selenium.run_js(
+            """
+            return pyodide.runPython(`
+                from js import Object
+                from pyodide import to_js
+                res = to_js({ x : x + 2 for x in range(5)}, dict_converter=Object.fromEntries)
+                res
+            `);
+            """
+        )
+        == {"0": 2, "1": 3, "2": 4, "3": 5, "4": 6}
+    )
+
+    assert (
+        selenium.run_js(
+            """
+            let px = pyodide.runPython("{ x : x + 2 for x in range(5)}");
+            let result = px.toJs({dict_converter : Object.fromEntries});
+            px.destroy();
+            return result;
+            """
+        )
+        == {"0": 2, "1": 3, "2": 4, "3": 5, "4": 6}
+    )
+
+    selenium.run("del res; del arrayFrom; del Object")
+
+
 def test_python2js_long_ints(selenium):
     assert selenium.run("2**30") == 2 ** 30
     assert selenium.run("2**31") == 2 ** 31