internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: onnx/onnx

Issue:
mypy typecheck (mypy version now is 0.760 in main branch) cannot work with Python 3.8 and 3.9 
# Bug Report
mypy typecheck (mypy 0.760 in main branch) cannot work with Python 3.8 and 3.9. It will throw few check errors. It is not caught by CI (AzurePipeline) because it only uses Python 3.6 and 3.7 with mypy typecheck. We should add 3.8 and 3.9 in AzurePipeline to cover it.

### Reproduction instructions
```
python setup.py typecheck
...
```

### Expected behavior
Should not produce any error.


Discussion:

Resolving patch:
diff --git a/onnx/helper.py b/onnx/helper.py
--- a/onnx/helper.py
+++ b/onnx/helper.py
@@ -826,10 +826,10 @@ def printable_node(node: NodeProto, prefix: Text = '', subgraphs: bool = False)
     printed_attrs = []
     for attr in node.attribute:
         if subgraphs:
-            printed_attr, gs = printable_attribute(attr, subgraphs)
-            assert isinstance(gs, list)
-            graphs.extend(gs)
-            printed_attrs.append(printed_attr)
+            printed_attr_subgraphs = printable_attribute(attr, subgraphs)
+            assert isinstance(printed_attr_subgraphs[1], list)
+            graphs.extend(printed_attr_subgraphs[1])
+            printed_attrs.append(printed_attr_subgraphs[0])
         else:
             printed = printable_attribute(attr)
             assert isinstance(printed, Text)
@@ -894,10 +894,10 @@ def printable_graph(graph: GraphProto, prefix: Text = '') -> Text:
     graphs: List[GraphProto] = []
     # body
     for node in graph.node:
-        pn, gs = printable_node(node, indent, subgraphs=True)
-        assert isinstance(gs, list)
-        content.append(pn)
-        graphs.extend(gs)
+        contents_subgraphs = printable_node(node, indent, subgraphs=True)
+        assert isinstance(contents_subgraphs[1], list)
+        content.append(contents_subgraphs[0])
+        graphs.extend(contents_subgraphs[1])
     # tail
     tail = ['return']
     if len(graph.output):
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -327,7 +327,7 @@ def run(self):
 tests_require.append('nbval')
 tests_require.append('tabulate')
 
-extras_require['mypy'] = ['mypy==0.760', 'types-protobuf==3.18.4']
+extras_require['mypy'] = ['mypy==0.782', 'types-protobuf==3.18.4']
 
 ################################################################################
 # Final


Test patch:
diff --git a/onnx/backend/test/report/coverage.py b/onnx/backend/test/report/coverage.py
--- a/onnx/backend/test/report/coverage.py
+++ b/onnx/backend/test/report/coverage.py
@@ -159,6 +159,7 @@ def report_csv(self, all_ops: List[Text], passed: List[Optional[Text]], experime
         if os.path.isfile(nodes_path):
             with open(nodes_path, 'r') as nodes_file:
                 reader = csv.DictReader(nodes_file)
+                assert reader.fieldnames
                 frameworks = list(reader.fieldnames)
                 for row in reader:
                     op = row[str('Op')]
Repository: onnx/onnx

Issue:
Is this concat operation supported by ONNX?
# Source
PaddlePaddle->onnx->onnx-sim 
onnx version:11.0
# Describe
The input feature graph and the two learned feature graphs are spliced in dimension.
# New Operator
![concat](https://user-images.githubusercontent.com/37217594/149057480-b93da9a4-b261-46d9-8ee5-f8b3f03e0906.png)
# one of the two learned feature graphs info:
![a35e8db6072a06bf6ffb120c7e5df03](https://user-images.githubusercontent.com/37217594/149057924-66e281ad-e21f-4e30-84f2-d27a83da0529.png)
# Confused point
How does this work? 
Any suggestions if I need to refactor it with traditional operations by caffe？

# Thanks

I hope I can get your help！
Thanks ! 





Discussion:
Hi @tensorflowt,
Yes Conat is a supported ONNX operator https://github.com/onnx/onnx/blob/main/docs/Operators.md#concat. You can check all supported operators in the same document. As you can see, the output shape of concat_8.tmp_0 is 1x130x76x76.
@tensorflowt: I don't understand your question. Is it simply - Is concat supported in onnx? or do you need some help for converting the model to onnx?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -318,7 +318,12 @@ def run(self):
 tests_require.append('nbval')
 tests_require.append('tabulate')
 
-extras_require['mypy'] = ['mypy==0.782', 'types-protobuf==3.18.4']
+extras_require['lint'] = [
+    'clang-format==13.0.0',
+    'flake8',
+    'mypy==0.782',
+    'types-protobuf==3.18.4',
+]
 
 ################################################################################
 # Final


Test patch:
Repository: onnx/onnx

Issue:
Make C++ and Python checker API consistent
Python checker API supports `full_check` arg:
https://github.com/onnx/onnx/blob/fa6f8cfdce3d86346e8a7494f3062b98416c85fb/onnx/checker.py#L94

C++ does not.
It'd be nice for them to be consistent.


Discussion:
Hello, do you mind if I work on this issue?

If I understand correctly, the idea is to add the `full_check` argument (which enables shape inference) to the `check_model` functions of the C++ API (checker.h/checker.cc).
Yes that's the request. Feel free to work on it. Thanks!

Resolving patch:
diff --git a/onnx/checker.py b/onnx/checker.py
--- a/onnx/checker.py
+++ b/onnx/checker.py
@@ -94,18 +94,14 @@ def check_model(model: Union[ModelProto, str, bytes], full_check: bool = False)
     """
     # If model is a path instead of ModelProto
     if isinstance(model, str):
-        C.check_model_path(model)
-        if full_check:
-            onnx.shape_inference.infer_shapes_path(model, check_type=True, strict_mode=True)
+        C.check_model_path(model, full_check)
     else:
         protobuf_string = model if isinstance(model, bytes) else model.SerializeToString()
         # If the protobuf is larger than 2GB,
         # remind users should use the model path to check
         if sys.getsizeof(protobuf_string) > MAXIMUM_PROTOBUF:
             raise ValueError('This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.')
-        C.check_model(protobuf_string)
-        if full_check:
-            onnx.shape_inference.infer_shapes(model, check_type=True, strict_mode=True)
+        C.check_model(protobuf_string, full_check)
 
 
 ValidationError = C.ValidationError


Test patch:
Repository: onnx/onnx

Issue:
[Documentation] Publish md files under docs/ to the documentation site
https://github.com/onnx/onnx/tree/main/docs has a lot of good information about onnx. It would be nice if we can publish them all to the documentation site for better discoverability and readability. 


Discussion:
https://github.com/executablebooks/rst-to-myst

Resolving patch:
diff --git a/docs/docsgen/source/onnx_sphinx.py b/docs/docsgen/source/onnx_sphinx.py
--- a/docs/docsgen/source/onnx_sphinx.py
+++ b/docs/docsgen/source/onnx_sphinx.py
@@ -9,6 +9,7 @@
 import os
 import pathlib
 import re
+import shutil
 import sys
 import textwrap
 from typing import Any
@@ -21,6 +22,18 @@
 from onnx.backend.test.case.base import _Exporter
 from onnx.defs import OpSchema
 
+REPO_DOCS_EXCLUDE = {
+    "Changelog-ml.md",
+    "Changelog.md",
+    "CIPipelines.md",
+    "CONTRIBUTING.md",
+    "Operators-ml.md",
+    "Operators.md",
+    "Relicensing.md",
+    "TestCoverage-ml.md",
+    "TestCoverage.md",
+}
+
 
 def _get_diff_template():
     return jinja2.Template(
@@ -845,6 +858,23 @@ def _generate_op_doc(app):
     onnx_documentation_folder(folder, flog=logger.info, max_opsets=max_opsets)
 
 
+def _copy_repo_docs(app):
+    logger = logging.getLogger(__name__)
+    dest_name = app.config.onnx_md_folder
+
+    docs_dir = pathlib.Path(__file__).parent.parent.parent  # docs
+    dest_folder = docs_dir / "docsgen" / "source" / dest_name
+    dest_folder.mkdir(parents=True, exist_ok=True)
+    # Copy all the markdown files from the folder except for the blocklisted ones
+
+    logger.info("Copying Markdown files from '%s' to '%s'", docs_dir, dest_folder)
+    for file in docs_dir.glob("*.md"):
+        if file.name in REPO_DOCS_EXCLUDE:
+            continue
+        shutil.copy(file, dest_folder)
+        logger.info("Copying '%s'", file.name)
+
+
 def setup(app):
     """
     Sphinx extension `onnx_sphinx` displays documentation
@@ -853,8 +883,11 @@ def setup(app):
     import sphinx
 
     app.add_config_value("onnx_doc_folder", "operators", "env")
+    # Folder for storing the Markdown documentation from the repository
+    app.add_config_value("onnx_md_folder", "repo-docs", "env")
     app.add_config_value("max_opsets", {}, "env")
     app.connect("builder-inited", _generate_op_doc)
+    app.connect("builder-inited", _copy_repo_docs)
     return {"version": sphinx.__display_version__, "parallel_read_safe": True}
 
 


Test patch:
Repository: onnx/onnx

Issue:
Refine docs for check_model
Current version:

> Check the consistency of a model. An exception is raised if the test fails.

It would be good if we document the kind of checks done and the type of exception raised so users know what to catch for; as well as clarify that it also runs shape inference when strict is True. (Right now it says `if True, the function checks shapes can be inferred`）

Should we default `strict` to `True`? @jcwchen 


Discussion:
> Should we default strict to True? @jcwchen

I guess you are saying full_check in onnx.checker.check_model? IIRC, we didn't set full_check by default due to compatibility. Previously onnx.checker.check_model does not check anything related to shape/type inference. Such a check was brought after https://github.com/onnx/onnx/pull/2367 (>=ONNX 1.7.0). Enabling full_check by default might cause some old models fail with onnx.checker, although they might be still runnable by runtime.
IMHO, it should not be considered a breaking change if a checker becomes better and gains more capabilities over time. Changing the default to fail on an incomplete check (e.g. missing type/shape inference for an Op) might be quite disruptive, though.

Resolving patch:
diff --git a/onnx/checker.py b/onnx/checker.py
--- a/onnx/checker.py
+++ b/onnx/checker.py
@@ -124,11 +124,20 @@ def check_model(
     full_check: bool = False,
     skip_opset_compatibility_check: bool = False,
 ) -> None:
-    """Check the consistency of a model. An exception is raised if the test fails.
+    """Check the consistency of a model.
+
+    An exception will be raised if the model's ir_version is not set
+    properly or is higher than checker's ir_version, or if the model
+    has duplicate keys in metadata_props.
+
+    If IR version >= 3, the model must specify opset_import.
+    If IR version < 3, the model cannot have any opset_import specified.
 
     Args:
-        model: Model to check.
-        full_check: If True, the function also checks for shapes that can be inferred.
+        model: Model to check. If model is a path, the function checks model
+            path first. If the model bytes size is larger than 2GB, function
+            should be called using model path.
+        full_check: If True, the function also runs shape inference check.
         skip_opset_compatibility_check: If True, the function skips the check for
             opset compatibility.
     """


Test patch:
Repository: onnx/onnx

Issue:
check_function requires contexts as arguments which breaks backward compatibility
https://github.com/onnx/onnx/pull/5693 added required parameters to the `check_function` function in checker which breaks backward compatibility. Should we provide default contexts to `check_function` as well?




Discussion:

Resolving patch:
diff --git a/onnx/checker.py b/onnx/checker.py
--- a/onnx/checker.py
+++ b/onnx/checker.py
@@ -84,37 +84,37 @@ def check_tensor(tensor: TensorProto, ctx: C.CheckerContext = DEFAULT_CONTEXT) -
 def check_attribute(
     attr: AttributeProto,
     ctx: C.CheckerContext = DEFAULT_CONTEXT,
-    lex_ctx: C.LexicalScopeContext = LEXICAL_SCOPE_CONTEXT,
+    lexical_scope_ctx: C.LexicalScopeContext = LEXICAL_SCOPE_CONTEXT,
 ) -> None:
     _ensure_proto_type(attr, AttributeProto)
-    return C.check_attribute(attr.SerializeToString(), ctx, lex_ctx)
+    return C.check_attribute(attr.SerializeToString(), ctx, lexical_scope_ctx)
 
 
 def check_node(
     node: NodeProto,
     ctx: C.CheckerContext = DEFAULT_CONTEXT,
-    lex_ctx: C.LexicalScopeContext = LEXICAL_SCOPE_CONTEXT,
+    lexical_scope_ctx: C.LexicalScopeContext = LEXICAL_SCOPE_CONTEXT,
 ) -> None:
     _ensure_proto_type(node, NodeProto)
-    return C.check_node(node.SerializeToString(), ctx, lex_ctx)
+    return C.check_node(node.SerializeToString(), ctx, lexical_scope_ctx)
 
 
 def check_function(
     function: FunctionProto,
-    ctx: C.CheckerContext,
-    lex_ctx: C.LexicalScopeContext,
+    ctx: C.CheckerContext = DEFAULT_CONTEXT,
+    lexical_scope_ctx: C.LexicalScopeContext = LEXICAL_SCOPE_CONTEXT,
 ) -> None:
     _ensure_proto_type(function, FunctionProto)
-    C.check_function(function.SerializeToString(), ctx, lex_ctx)
+    C.check_function(function.SerializeToString(), ctx, lexical_scope_ctx)
 
 
 def check_graph(
     graph: GraphProto,
     ctx: C.CheckerContext = DEFAULT_CONTEXT,
-    lex_ctx: C.LexicalScopeContext = LEXICAL_SCOPE_CONTEXT,
+    lexical_scope_ctx: C.LexicalScopeContext = LEXICAL_SCOPE_CONTEXT,
 ) -> None:
     _ensure_proto_type(graph, GraphProto)
-    return C.check_graph(graph.SerializeToString(), ctx, lex_ctx)
+    return C.check_graph(graph.SerializeToString(), ctx, lexical_scope_ctx)
 
 
 def check_sparse_tensor(


Test patch:
Repository: google/pytype

Issue:
python -m pip install -U . doesn't work.
It ought to be possible to install pytype using pip by running

```
python -m pip install -U .
```

but doing so causes an error message.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,8 @@ def scan_package_data(path, pattern):
              ],
     scripts=['scripts/pytype', 'scripts/pytd'],
     package_data={'pytype': ['pytd/builtins/*',
-                             'pytd/stdlib/*',
+                             'pytd/stdlib/os/*.pytd',
+                             'pytd/stdlib/*.pytd',
                             ] + typeshed},
     requires=['ply (>=3.4)'],
     install_requires=['ply>=3.4'],


Test patch:
Repository: google/pytype

Issue:
Allow pytype to override an existing logging level.
When pytype is called from analyze_project, we need to be able to
override analyze_project's own logging level in order to use the one
requrested for pytype.

This will have to go through the github->internal->github workflow, but I figured that it would be nice to make open-source-related changes on GitHub.


Discussion:
I don't understand the motivation for this change. Why do analyze_project and pytype have different logging levels in the first place?
Because we often want to use a less noisy logging level for pytype than for analyze_project. For example, when I request logging.INFO messages for analyze_project, I want to see things like the configuration options from setup.cfg, not the entire bytecode execution flow for every file.
Ah, I see! That makes sense.

I believe GitHub is a bit behind HEAD right now, so I'd like to sync before submitting this.

Resolving patch:
diff --git a/pytype/config.py b/pytype/config.py
--- a/pytype/config.py
+++ b/pytype/config.py
@@ -408,7 +408,11 @@ def _store_verbosity(self, verbosity):
       # "verbosity=-1" can be used to disable all logging, so configure
       # logging accordingly.
       basic_logging_level = logging.CRITICAL + 1
-    logging.basicConfig(level=basic_logging_level)
+    if logging.root.handlers:
+      # When calling pytype as a library, override the caller's logging level.
+      logging.root.setLevel(basic_logging_level)
+    else:
+      logging.basicConfig(level=basic_logging_level)
 
   def _store_pythonpath(self, pythonpath):
     # Note that the below gives [""] for "", and ["x", ""] for "x:"


Test patch:
Repository: google/pytype

Issue:
Future-proof preconditions.py:_TOKEN_RE
3.7 adds a warning about possible future changes to re: https://bugs.python.org/issue30349
A future version of python will add nested sets, which allows nesting of sets using `[...]`. Escape the inner `[` in the re so it doesn't trigger a nested set.
Closes #140.


Discussion:
Would like to resolve #143 before importing this one.

Resolving patch:
diff --git a/pytype/pytd/parse/preconditions.py b/pytype/pytd/parse/preconditions.py
--- a/pytype/pytd/parse/preconditions.py
+++ b/pytype/pytd/parse/preconditions.py
@@ -163,7 +163,7 @@ def allowed_types(self):
 
 # RE to match a single token.  Leading whitepace is ignored.
 _TOKEN_RE = re.compile(
-    r"\s*(?:(?P<literal>[[\]{}])|(?P<word>[a-zA-Z_]\w*))")
+    r"\s*(?:(?P<literal>[\[\]{}])|(?P<word>[a-zA-Z_]\w*))")
 
 # Token codes (aside from literal characters)
 _TOKEN_NAME = 1


Test patch:
Repository: google/pytype

Issue:
Add test_data to MANIFEST.in
This PR also needs to be imported and re-exported rather than merged directly. I'm planning to use this one to test the import process fix I sent you.

Fixes https://github.com/google/pytype/issues/245.


Discussion:

Resolving patch:
diff --git a/pytype/__version__.py b/pytype/__version__.py
--- a/pytype/__version__.py
+++ b/pytype/__version__.py
@@ -1,2 +1,2 @@
 # pylint: skip-file
-__version__ = '2019.02.13'
+__version__ = '2019.03.01'


Test patch:
Repository: google/pytype

Issue:
Add __init__ files to two tools modules
These two modules were not declared as modules because their `__init__.py` is missing. Python 3+ compensates for this by considering them namespaces, but the `annotate-ast` command (and probably a few others) are broken on Python 2.7 due to lack of an `__init__.py`

Here is an MWE (it assumes you have `virtualenv` installed):

```
cd $(mktemp -d)
virtualenv venv --python=`which python2.7`
source venv/bin/activate
pip install pytype
hash -r  # Ensure that annotate-ast points to the python2.7 version
annotate-ast a.py
```

Prior to this PR, this should fail with some message like this:

```
$ annotate-ast
Traceback (most recent call last):
  File "/tmp/tmp.VKZgcka8q6/venv/bin/annotate-ast", line 5, in <module>
    from pytype.tools.annotate_ast.main import main
ImportError: No module named annotate_ast.main
```

I have not added a test for this, but I suspect that #376, combined with moving the tests into a dedicated `tests/` directory (rather than in-line, as they are now), will catch this and many other issues like it.


Discussion:
This has apparently started failing on Python 2.7 because the `__init__.py` is triggering `pylint` to actually lint the files, and `pytype.tools.annotate_ast.main` contains a print statement with parens around it. It appears that [the file is intended to be linted as Python 3](https://github.com/google/pytype/blob/a735b1109229cacfcbbf183d1ea85d2c31065d70/pytype/tools/annotate_ast/main.py#L1), but `pylint` doesn't know that, so I think the options are:

1. Add a `from __future__ import print_function` statement at the top of the file.
2. Explicitly ignore the file in [`travis_script.py`](https://github.com/google/pytype/blob/a735b1109229cacfcbbf183d1ea85d2c31065d70/build_scripts/travis_script.py#L45) as is being done for `xref`.

I would lean towards the first one as long as the package supports Python 2 anyway.
Either option sounds fine. 1) is probably simpler, since I vaguely recall having to change multiple files to exclude xref from Python 2 linting.
@rchen152 Great, I went with the first one and it seems to have fixed the linting issue. Thanks

Resolving patch:
diff --git a/pytype/tools/annotate_ast/__init__.py b/pytype/tools/annotate_ast/__init__.py
new file mode 100644
diff --git a/pytype/tools/annotate_ast/main.py b/pytype/tools/annotate_ast/main.py
--- a/pytype/tools/annotate_ast/main.py
+++ b/pytype/tools/annotate_ast/main.py
@@ -1,6 +1,8 @@
 # python3
 """Call annotate_ast on a source file."""
 
+from __future__ import print_function
+
 import argparse
 import sys
 
diff --git a/pytype/tools/traces/__init__.py b/pytype/tools/traces/__init__.py
new file mode 100644


Test patch:
Repository: google/pytype

Issue:
Use list comprehensions for flattening
This PR changes the visitor and typegraph utils to use list comprehensions for flattening nested sequences instead of `sum([...], [])` which can be slow for large inputs. This should improve readability and also slightly improve performance.


Discussion:

Resolving patch:
diff --git a/pytype/pytd/visitors.py b/pytype/pytd/visitors.py
--- a/pytype/pytd/visitors.py
+++ b/pytype/pytd/visitors.py
@@ -1235,9 +1235,9 @@ def VisitFunction(self, f):
       Function with the new signatures.
     """
 
-    # concatenate return value(s) from VisitSignature
-    signatures = sum([ExpandSignature(s) for s in f.signatures], [])
-    return f.Replace(signatures=tuple(signatures))
+    # flatten return value(s) from VisitSignature
+    signatures = tuple(ex for s in f.signatures for ex in ExpandSignature(s))  # pylint: disable=g-complex-comprehension
+    return f.Replace(signatures=signatures)
 
 
 class AdjustTypeParameters(Visitor):
diff --git a/pytype/typegraph/cfg_utils.py b/pytype/typegraph/cfg_utils.py
--- a/pytype/typegraph/cfg_utils.py
+++ b/pytype/typegraph/cfg_utils.py
@@ -105,8 +105,8 @@ def _deep_values_list_product(values_list, seen, complexity_limit):
   """Take the deep Cartesian product of a list of list of Values."""
   result = []
   for row in itertools.product(*(values for values in values_list if values)):
-    extra_params = sum([entry.data.unique_parameter_values()
-                        for entry in row if entry not in seen], [])
+    extra_params = [value for entry in row if entry not in seen  # pylint: disable=g-complex-comprehension
+                    for value in entry.data.unique_parameter_values()]
     extra_values = (extra_params and
                     _deep_values_list_product(extra_params, seen.union(row),
                                               complexity_limit))


Test patch: