internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: opsdroid/opsdroid

Issue:
Looking for Teams Service Endpoints in Mongo database forces error when no endpoints found within specified Mongo collection
Hi all,

So I'm getting an error when using both teams and a mongo database. The database uses the new SRV protocol which has been added recently, much thanks to @jacobtomlinson  and @FabioRosado  for helping by adding functionality for that. The teams connector also goes through the new developer portal. So my bot and app are done through that versus the former app studio method they used before.

I'm also doing my opsdroid deployment through docker swarm. 

For some reason Microsoft teams is looking for teams service endpoints within the specified collection I use within my config.
The error pasted below errors out when opsdroid is looking for those endpoints but can't find them within the mongodb collection.

![opsdroid error print out](https://user-images.githubusercontent.com/7581178/141191140-8361611c-a1ba-45cc-80c9-0ad519017b36.png)



Discussion:
I'm not exactly sure what might be causing the error, if it's something teams related due to the new developer portal or mongo-related.
Ah this is interesting. This is happening because connectors are started before databases, and teams is trying to use a database during startup.

Resolving patch:
diff --git a/opsdroid/core.py b/opsdroid/core.py
--- a/opsdroid/core.py
+++ b/opsdroid/core.py
@@ -184,8 +184,8 @@ async def start(self):
         if len(self.skills) == 0:
             self.critical(_("No skills in configuration, at least 1 required"), 1)
 
+        await self.start_databases()
         await self.start_connectors()
-        self.create_task(self.start_databases())
         self.create_task(self.watch_paths())
         self.create_task(parse_crontab(self))
         self.create_task(self.web_server.start())


Test patch:
Repository: pyodide/pyodide

Issue:
Make work on Chrome

Make work on Chrome



Discussion:
Reopening.  While this now works for the MAIN_MODULE, it's failing for SIDE_MODULES still, with no obvious resolution yet.
Reopening.  While this now works for the MAIN_MODULE, it's failing for SIDE_MODULES still, with no obvious resolution yet.

Resolving patch:
diff --git a/tools/buildpkg.py b/tools/buildpkg.py
--- a/tools/buildpkg.py
+++ b/tools/buildpkg.py
@@ -144,7 +144,8 @@ def package_files(buildpath, srcpath, pkg, args):
         '--js-output={}'.format(os.path.join(buildpath, name + '.js')),
         '--export-name=pyodide',
         '--exclude', '*.wasm.pre',
-        '--exclude', '__pycache__'], check=True)
+        '--exclude', '__pycache__',
+        '--use-preload-plugins'], check=True)
     subprocess.run([
         'uglifyjs',
         os.path.join(buildpath, name + '.js'),


Test patch:
Repository: pyodide/pyodide

Issue:
Improve parsing of result line
The parsing of the input Python to find the last line which will be evaluated (rather than executed) to provide the result is probably a little brittle in certain corner cases.  We should look at what IPython does here and copy that.


Discussion:

Resolving patch:
diff --git a/src/pyodide.py b/src/pyodide.py
--- a/src/pyodide.py
+++ b/src/pyodide.py
@@ -4,6 +4,7 @@
 
 from js import XMLHttpRequest
 
+import ast
 import io
 
 
@@ -17,4 +18,23 @@ def open_url(url):
     return io.StringIO(req.response)
 
 
-__all__ = ['open_url']
+def eval_code(code, ns):
+    """
+    Runs a string of code, the last part of which may be an expression.
+    """
+    mod = ast.parse(code)
+    if isinstance(mod.body[-1], ast.Expr):
+        expr = ast.Expression(mod.body[-1].value)
+        del mod.body[-1]
+    else:
+        expr = None
+
+    if len(mod.body):
+        exec(compile(mod, '<exec>', mode='exec'), ns, ns)
+    if expr is not None:
+        return eval(compile(expr, '<eval>', mode='eval'), ns, ns)
+    else:
+        return None
+
+
+__all__ = ['open_url', 'eval_code']


Test patch:
Repository: pyodide/pyodide

Issue:
Full build path is included in package `.js` files
As @rth pointed out in #121, the full build path to the `.data` file is included in the `.js` file for each package.  This is *really* a problem, since it doesn't prevent the packages from being deployed anywhere, but it is leaking information we probably don't want to and makes the builds less reproducible.


Discussion:

Resolving patch:
diff --git a/tools/buildpkg.py b/tools/buildpkg.py
--- a/tools/buildpkg.py
+++ b/tools/buildpkg.py
@@ -143,14 +143,15 @@ def package_files(buildpath, srcpath, pkg, args):
     subprocess.run([
         'python',
         Path(os.environ['EMSCRIPTEN']) / 'tools' / 'file_packager.py',
-        buildpath / (name + '.data'),
+        name + '.data',
         '--preload',
         '{}@/lib/python3.6/site-packages'.format(libdir),
-        '--js-output={}'.format(buildpath / (name + '.js')),
+        '--js-output={}'.format(name + '.js'),
         '--export-name=pyodide',
         '--exclude', '*.wasm.pre',
         '--exclude', '__pycache__',
-        '--use-preload-plugins'], check=True)
+        '--use-preload-plugins'],
+        cwd=buildpath, check=True)
     subprocess.run([
         'uglifyjs',
         buildpath / (name + '.js'),


Test patch:
Repository: pyodide/pyodide

Issue:
ValueError: invalid __array_struct__ when using js arrays of arrays and numpy
When using a matrix (array of array of numbers) in javascript and trying to convert that to a numpy array, it fails with the error `ValueError: invalid __array_struct__`

To reproduce:
JavaScript:
```
window.A = [[1,2,3],[4,5,6]];
```
Python:
```
import numpy
from js import A
m = numpy.array(A)
```


Discussion:
Interestingly, other libraries besides numpy do seem to be able to work with JavaScript matrices. For example, statistics does work with a JavaScript matrix.
JavaScript:
```
window.A = [[1,2,3],[4,5,6]];
```
Python:
```
import statistics
from js import A
statistics.mean(A[0])
```
After some more digging around in the code, it appears `__array_struct__` is a special `numpy` thing.
Looking in the code for the numpy package at `packages/numpy`, I see that it is always accessed using the function `PyArray_LookupSpecial_OnInstance`. When I look at that function, it has the interesting comment: 
```
 * Implements incorrect special method lookup rules, that break the python
 * convention, and looks on the instance, not the type.
 *
 * Kept for backwards compatibility. In future, we should deprecate this.
```
This seems like it could be the root of the problem, but it is unclear to me what change should be made. Patching the implementation of `PyArray_LookupSpecial_OnInstance` to be that of `PyArray_LookupSpecial` might make it work for JavaScript objects, but could break it for native python objects. It's very hard to follow how this code works.
More searching the code reveals that the error must be produced at line 2241 of `packages/numpy/build/numpy-1.15.1/numpy/core/src/multiarray/ctors.c`. This means that the call to `PyArray_LookupSpecial_OnInstance` at line 2201 of `ctors.c` must be returning something which is not `NULL`, but is also not a valid `__array_struct__`. Still unclear to me how/why this is happening.
I can't reproduce this as of Pyodide 0.8.2 -- it's possible the changes to object coercion helped.

However, even given that, I wouldn't expect Numpy to be able to automatically convert this given how Javascript Arrays and Objects are kinda-sorta the same thing.  I think the best we could expect is to require:

```
np.array([list(x) for x in A])
```

(and maybe provide as `aslist` helper function to handle nested lists).

Resolving patch:
diff --git a/src/pyodide.py b/src/pyodide.py
--- a/src/pyodide.py
+++ b/src/pyodide.py
@@ -67,4 +67,16 @@ def find_imports(code):
     return list(imports)
 
 
-__all__ = ['open_url', 'eval_code', 'find_imports']
+def as_nested_list(obj):
+    """
+    Assumes a Javascript object is made of (possibly nested) arrays and
+    converts them to nested Python lists.
+    """
+    try:
+        it = iter(obj)
+        return [as_nested_list(x) for x in it]
+    except TypeError:
+        return obj
+
+
+__all__ = ['open_url', 'eval_code', 'find_imports', 'as_nested_list']


Test patch:
Repository: pyodide/pyodide

Issue:
Decimal is 450x slower than native
The Decimal data type performs 450 times slower in pyodide than the same code run natively. 

To test this I replaced the benchmarks with two of my own. The first I called `large_list`, and it multiplies two 1000 item lists of floats. It runs a very reasonable 10x slower than native. The second test is identical to the first, except that instead of floats it uses Decimals. It runs a painful 450x slower than native.
Here is `large_list`:
```
# setup: A = [2.1 for i in range(1000)] ; B = [3.2 for i in range(1000)]
# run: large_list(A, B)

def large_list(A, B):
    return [a*b for a,b in zip(A,B)]
```
Here is `large_decimal_list`:
```
# setup: from decimal import Decimal ; A = [Decimal('2.1') for i in range(1000)] ; B = [Decimal('3.2') for i in range(1000)]
# run: large_decimal_list(A, B)

def large_decimal_list(A, B):
    return [a*b for a,b in zip(A,B)]
```
And here is the resulting `benchmarks.json` file:
```
{
  "pystone": {"native": 0.456669, "firefox": 2.843, "chrome": 2.95},
  "large_decimal_list": {"native": 0.004931997666264781, "firefox": 2.188, "chrome": 2.229016110777778},
  "large_list": {"native": 0.002595901883776403, "firefox": 0.0214444444444444, "chrome": 0.025382222222222122}
}
```

When I profile similar code in the browser, I see that 95% of the total time (27% self time) is spent in one function (__PyEval_EvalFrameDefault), but I don't know how to proceed on debugging this further. I looked through the Decimal code at 
pyodide/cpython/build/3.7.0/host/lib/python3.7/_pydecimal.py, and at the PyEval_EvalFrameDefault code at pyodide/cpython/build/3.7.0/host/Python-3.7.0/Python/ceval.c  but nothing jumped out at me as a likely culprit.



Discussion:
It looks like the C implementation of decimal isn't being built, so it's falling back to the Python implementation, which is understandably much slower.

The process would basically involve adding the appropriate incantations to `Setup.local`.

Resolving patch:
diff --git a/benchmark/benchmarks/large_decimal_list.py b/benchmark/benchmarks/large_decimal_list.py
new file mode 100644
--- /dev/null
+++ b/benchmark/benchmarks/large_decimal_list.py
@@ -0,0 +1,6 @@
+# setup: from decimal import Decimal ; A = [Decimal('2.1') for i in range(1000)] ; B = [Decimal('3.2') for i in range(1000)]  # noqa
+# run: large_decimal_list(A, B)
+
+
+def large_decimal_list(A, B):
+    return [a * b for a, b in zip(A, B)]


Test patch:
Repository: pyodide/pyodide

Issue:
A more reasonable error when importing js
I realize that only `from js import something` is supported, but a more reasonable error message for `import js` would be very helpful:
```
import js
Error: TypeError: object of type 'NoneType' has no len()

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/lib/python3.7/site-packages/pyodide.py", line 42, in eval_code
    exec(compile(mod, '<exec>', mode='exec'), ns, ns)
  File "<exec>", line 1, in <module>
SystemError: <JsImport object at 0x81c2e8> returned a result with an error set
```


Discussion:

Resolving patch:
diff --git a/packages/matplotlib/src/wasm_backend.py b/packages/matplotlib/src/wasm_backend.py
--- a/packages/matplotlib/src/wasm_backend.py
+++ b/packages/matplotlib/src/wasm_backend.py
@@ -103,10 +103,10 @@ def get_dpi_ratio(self, context):
     def create_root_element(self):
         # Designed to be overridden by subclasses for use in contexts other
         # than iodide.
-        from js import iodide
-        if iodide is not None:
+        try:
+            from js import iodide
             return iodide.output.element('div')
-        else:
+        except ImportError:
             return document.createElement('div')
 
     def show(self):


Test patch:
diff --git a/test/test_python.py b/test/test_python.py
--- a/test/test_python.py
+++ b/test/test_python.py
@@ -303,11 +303,18 @@ def test_typed_arrays(selenium, wasm_heap, jstype, pytype):
 def test_import_js(selenium):
     result = selenium.run(
         """
-        from js import window
-        window.title = 'Foo'
-        window.title
+        import js
+        js.window.title = 'Foo'
+        js.window.title
         """)
     assert result == 'Foo'
+    result = selenium.run(
+        """
+        dir(js)
+        """)
+    assert len(result) > 100
+    assert 'document' in result
+    assert 'window' in result
 
 
 def test_pyimport_multiple(selenium):
Repository: pyodide/pyodide

Issue:
Typo?
Wondering what "wv" means or if this should read "more complex things."

https://github.com/iodide-project/pyodide/blob/163ab43b64180223d010cdcdcdecd17307cc5a45/pyodide_build/mkpkg.py#L77-L79


Discussion:
Thanks! Yes, I think so. Don't hesitate to open a PR to fix it...

Resolving patch:
diff --git a/pyodide_build/mkpkg.py b/pyodide_build/mkpkg.py
--- a/pyodide_build/mkpkg.py
+++ b/pyodide_build/mkpkg.py
@@ -75,7 +75,7 @@ def make_package(package, version=None):
 def make_parser(parser):
     parser.description = '''
 Make a new pyodide package. Creates a simple template that will work
-for most pure Python packages, but will have to be edited for more wv
+for most pure Python packages, but will have to be edited for more
 complex things.'''.strip()
     parser.add_argument(
         'package', type=str, nargs=1,


Test patch:
Repository: pyodide/pyodide

Issue:
Calling yaml.load() without Loader=... is deprecated
For each built packages there is now the following deprecation warning ,
```
pyodide_build/common.py:27: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  return yaml.load(fd)
```
it would be nice to fix this.


Discussion:

Resolving patch:
diff --git a/pyodide_build/common.py b/pyodide_build/common.py
--- a/pyodide_build/common.py
+++ b/pyodide_build/common.py
@@ -33,7 +33,7 @@ def parse_package(package):
 
     # TODO: Validate against a schema
     with open(package) as fd:
-        return yaml.load(fd)
+        return yaml.safe_load(fd)
 
 
 def _parse_package_subset(query: Optional[str]) -> Optional[Set[str]]:


Test patch:
Repository: pyodide/pyodide

Issue:
JsProxy does not handle Error correctly
Example:
```python
from js import Error
e = Error.new("Hi")
raise e
```
Raises: `TypeError: exceptions must derive from BaseException`.
This came up in https://github.com/iodide-project/pyodide/pull/880#pullrequestreview-555341317. I will try to fix this.


Discussion:

Resolving patch:
diff --git a/src/pyodide-py/pyodide/__init__.py b/src/pyodide-py/pyodide/__init__.py
--- a/src/pyodide-py/pyodide/__init__.py
+++ b/src/pyodide-py/pyodide/__init__.py
@@ -1,6 +1,13 @@
-from ._base import open_url, eval_code, find_imports, as_nested_list
+from ._base import open_url, eval_code, find_imports, as_nested_list, JsException
 from .console import get_completions
 
 __version__ = "0.15.0"
 
-__all__ = ["open_url", "eval_code", "find_imports", "as_nested_list", "get_completions"]
+__all__ = [
+    "open_url",
+    "eval_code",
+    "find_imports",
+    "as_nested_list",
+    "get_completions",
+    "JsException",
+]
diff --git a/src/pyodide-py/pyodide/_base.py b/src/pyodide-py/pyodide/_base.py
--- a/src/pyodide-py/pyodide/_base.py
+++ b/src/pyodide-py/pyodide/_base.py
@@ -1,6 +1,8 @@
 """
 A library of helper utilities for connecting Python to the browser environment.
 """
+# Added by C:
+# JsException (from jsproxy.c)
 
 import ast
 from io import StringIO
@@ -8,6 +10,16 @@
 from typing import Dict, List, Any
 
 
+class JsException(Exception):
+    """
+    A wrapper around a Javascript Error to allow the Error to be thrown in Python.
+    """
+
+    # This gets overwritten in jsproxy.c, it is just here for autodoc and humans
+    # reading this file.
+    pass
+
+
 def open_url(url: str) -> StringIO:
     """
     Fetches a given URL


Test patch:
diff --git a/src/tests/test_typeconversions.py b/src/tests/test_typeconversions.py
--- a/src/tests/test_typeconversions.py
+++ b/src/tests/test_typeconversions.py
@@ -271,3 +271,39 @@ class Point {
     with pytest.raises(WebDriverException, match=msg):
         selenium.run("point.y")
     assert selenium.run_js("return point.y;") is None
+
+
+def test_javascript_error(selenium):
+    msg = "JsException: Error: This is a js error"
+    with pytest.raises(WebDriverException, match=msg):
+        selenium.run(
+            """
+            from js import Error
+            err = Error.new("This is a js error")
+            err2 = Error.new("This is another js error")
+            raise err
+            """
+        )
+
+
+def test_javascript_error_back_to_js(selenium):
+    selenium.run_js(
+        """
+        window.err = new Error("This is a js error")
+        """
+    )
+    assert (
+        selenium.run(
+            """
+        from js import err
+        py_err = err
+        type(py_err).__name__
+        """
+        )
+        == "JsException"
+    )
+    assert selenium.run_js(
+        """
+        return pyodide.globals["py_err"] === err
+        """
+    )
Repository: pyodide/pyodide

Issue:
CLAPACK.js is missing from the dev branch
For some reason, I started to see the CLAPACK.js missing error when using pyodide from https://cdn.jsdelivr.net/pyodide/dev/full/:

```
Couldn't load package from URL https://cdn.jsdelivr.net/pyodide/dev/full/CLAPACK.js
```


Discussion:
This is due to #927, because CLAPACK is now listed as a dependency of
scipy, despite the fact that it is already statically linked.


Resolving patch:
diff --git a/pyodide_build/buildall.py b/pyodide_build/buildall.py
--- a/pyodide_build/buildall.py
+++ b/pyodide_build/buildall.py
@@ -220,11 +220,15 @@ def build_packages(packages_dir: Path, outputdir: Path, args) -> None:
         "import_name_to_package_name": {},
     }
 
+    libraries = [pkg.name for pkg in pkg_map.values() if pkg.library]
+
     for name, pkg in pkg_map.items():
         if pkg.library:
             continue
 
-        package_data["dependencies"][name] = pkg.dependencies
+        package_data["dependencies"][name] = [
+            x for x in pkg.dependencies if x not in libraries
+        ]
         for imp in pkg.meta.get("test", {}).get("imports", [name]):
             package_data["import_name_to_package_name"][imp] = name
 


Test patch:
Repository: pyodide/pyodide

Issue:
Add PYODIDE_MINIMAL build option
From the added documentation,

> Minimal pyodide build can be enabled by setting the `PYODIDE_MINIMAL`
environment variable.  For instance,
> ```            
> PYODIDE_MINIMAL=true PYODIDE_PACKAGES="micropip" make
> ```            
>                
> This will,
>  - not include freetype and libpng libraries (it won't be possible to build matplotlib)
>  - not include the jedi library, disabling auto-completion in iodide
> 
> As as a result the size will of the core pyodide binaries will be ~15% smaller.

Addresses two points from https://github.com/iodide-project/pyodide/issues/646

Before (master),
```
6,6M  pyodide.asm.data
310K  pyodide.asm.data.js
2,8M  pyodide.asm.js
 11M  pyodide.asm.wasm
 16K  pyodide.js
 16K  pyodide_dev.js

Total: 20.7 MB
```
after (this PR with PYODIDE_MINIMAL=true)
```
5,1M  build/pyodide.asm.data
124K  build/pyodide.asm.data.js
2,6M  build/pyodide.asm.js
9,9M  build/pyodide.asm.wasm
 16K  build/pyodide.js
 16K  build/pyodide_dev.js

Total: 17.7 MB
```

so it's not that different (14% less), but it's start. 

Draft PR for now, as I think I need to go in a bit more details through tests that are run in the minimal build CI job.


Discussion:

Resolving patch:
diff --git a/src/pyodide-py/pyodide/console.py b/src/pyodide-py/pyodide/console.py
--- a/src/pyodide-py/pyodide/console.py
+++ b/src/pyodide-py/pyodide/console.py
@@ -32,6 +32,6 @@ def get_completions(
         cursor = len(code)
     code = code[:cursor]
     interp = jedi.Interpreter(code, namespaces)
-    completions = interp.completions()
+    completions = interp.complete()
 
     return [x.name for x in completions]


Test patch:
diff --git a/packages/jedi/test_jedi.py b/packages/jedi/test_jedi.py
new file mode 100644
--- /dev/null
+++ b/packages/jedi/test_jedi.py
@@ -0,0 +1,22 @@
+def test_jedi(selenium_standalone):
+    selenium_standalone.load_package("jedi")
+    result = selenium_standalone.run(
+        """
+        import jedi
+        script = jedi.Script("import json\\njson.lo", path='example.py')
+        completions = script.complete(2, len('json.lo'))
+        [el.name for el in completions]
+        """
+    )
+    assert result == ["load", "loads"]
+
+
+def test_completions(selenium_standalone):
+    selenium_standalone.load_package("jedi")
+    result = selenium_standalone.run(
+        """
+        import pyodide
+        pyodide.get_completions('import sys\\nsys.v')
+        """
+    )
+    assert result == ["version", "version_info"]
diff --git a/src/tests/test_python.py b/src/tests/test_python.py
--- a/src/tests/test_python.py
+++ b/src/tests/test_python.py
@@ -176,16 +176,6 @@ def test_unknown_attribute(selenium):
     )
 
 
-def test_completions(selenium):
-    result = selenium.run(
-        """
-        import pyodide
-        pyodide.get_completions('import sys\\nsys.v')
-        """
-    )
-    assert result == ["version", "version_info"]
-
-
 def test_run_python_debug(selenium):
     assert selenium.run_js("return pyodide._module.runPythonDebug('1+1');") == 2
     assert selenium.run_js(