internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rotki/rotki

Issue:
Failed to read balance for asset USDT at Bitcoin.de
## Problem Definition

In the UI the Backend notified: Failed to read balance for asset USDT at Bitcoin.de. Please report this error.

## Logs

I cannot deliver any relevant logs. I am using Docker and shared /logs with the host system. There was a rotki.log created on the host system, but it is empty. The command "docker logs <container>" gave also no hint what went wrong. The config level was adjusted from "info" to "debug" in the rotki_config.json. 

### System Description

Operating system: Ubuntu 23.10 / Docker
Rotki version: 1.32.1



Discussion:
For logs: You would need to setup debug logs. Read here: https://rotki.readthedocs.io/en/latest/contribute.html#id2

But I think I can see why the error happens. Could you join our discord to chat more there? Could also send you a test binary to confirm a fix: https://discord.rotki.com/

Resolving patch:
diff --git a/rotkehlchen/exchanges/bitcoinde.py b/rotkehlchen/exchanges/bitcoinde.py
--- a/rotkehlchen/exchanges/bitcoinde.py
+++ b/rotkehlchen/exchanges/bitcoinde.py
@@ -54,6 +54,7 @@
     'etheur',
     'bsveur',
     'ltceur',
+    'usdteur',
     'xrpeur',
     'dogeeur',
     'soleur',
@@ -65,6 +66,7 @@
 )
 BITCOINDE_TO_WORLD_SYMBOLS = {
     'SOL': 'SOL-2',
+    'USDT': 'eip155:1/erc20:0xdAC17F958D2ee523a2206206994597C13D831ec7',
 }
 
 


Test patch:
Repository: semgrep/semgrep-rules

Issue:
Model.save() should call super().save()



Discussion:

Resolving patch:
diff --git a/python/django/db/model-save.py b/python/django/db/model-save.py
new file mode 100644
--- /dev/null
+++ b/python/django/db/model-save.py
@@ -0,0 +1,21 @@
+from django.db import models
+from django.template.defaultfilters import slugify
+
+class Post(models.Model):
+    title = models.CharField(max_length=255)
+    slug = models.SlugField(unique=True, max_length=255)
+
+    # ok
+    def save(self, *args, **kwargs):
+        if not self.slug:
+            self.slug = slugify(self.title)
+        super(Post, self).save(*args, **kwargs)
+
+class Bad(models.Model):
+    title = models.CharField(max_length=255)
+    slug = models.SlugField(unique=True, max_length=255)
+
+    # ruleid: django-db-model-save-super
+    def save(self, *args, **kwargs):
+        if not self.slug:
+            self.slug = slugify(self.title)
\ No newline at end of file


Test patch:
Repository: semgrep/semgrep-rules

Issue:
Registering different routes with the same endpoint name (the function name by default). This will raise a runtime error but could be linted sooner.



Discussion:
I dont think this is doable in Sgrep because we need to keep track of a state for all route definitions we see and then alert on duplicates
Could we approximate with something like

```
@app.route("...")
def $R(...):
    ...
...
@app.route("...")
def $R(...):
    ...
```

Resolving patch:
diff --git a/python/flask/same-handler-name.py b/python/flask/same-handler-name.py
new file mode 100644
--- /dev/null
+++ b/python/flask/same-handler-name.py
@@ -0,0 +1,17 @@
+from flask import Flask
+
+app = Flask(__name__)
+
+# ruleid: flask-duplicate-handler-name
+@app.route('/hello')
+def hello():
+    return 'hello'
+
+@app.route('/hi', methods=["POST"])
+def hello():
+  return 'hi'
+
+# ok
+@app.route('/howdy/:name')
+def howdy(name):
+  return f"""howdy {name}"""
\ No newline at end of file


Test patch:
Repository: semgrep/semgrep-rules

Issue:
avoid csrf_exempt decorator
only very occasionally is this ok


Discussion:

Resolving patch:
diff --git a/python/django/security/csrf_exempt.py b/python/django/security/csrf_exempt.py
new file mode 100644
--- /dev/null
+++ b/python/django/security/csrf_exempt.py
@@ -0,0 +1,7 @@
+from django.http import HttpResponse
+# ruleid: django-csrf_exempt
+from django.views.decorators.csrf import csrf_exempt
+
+@csrf_exempt
+def my_view(request):
+    return HttpResponse('Hello world')
\ No newline at end of file


Test patch:
Repository: semgrep/semgrep-rules

Issue:
Flask Deprecated APIs
Needs list



Discussion:

Resolving patch:
diff --git a/python/flask/deprecated-apis.py b/python/flask/deprecated-apis.py
new file mode 100644
--- /dev/null
+++ b/python/flask/deprecated-apis.py
@@ -0,0 +1,46 @@
+from flask import Flask, json_available, request, testing
+
+app = Flask(__name__)
+
+# ruleid: flask-deprecated-apis
+if json_available:
+    pass
+
+# ruleid: flask-deprecated-apis
+blueprint = request.module
+
+# ruleid: flask-deprecated-apis
+builder = testing.make_test_environ_builder(app)
+
+# ruleid: flask-deprecated-apis
+app.open_session(...)
+
+# ruleid: flask-deprecated-apis
+app.save_session(...)
+
+# ruleid: flask-deprecated-apis
+app.make_null_session(...)
+
+# ruleid: flask-deprecated-apis
+app.init_jinja_globals(...)
+
+# ruleid: flask-deprecated-apis
+app.request_globals_class(...)
+
+# ruleid: flask-deprecated-apis
+app.static_path(...)
+
+# ruleid: flask-deprecated-apis
+app.config.from_json(...)
+
+
+@app.route("/foo")
+def foo():
+    pass
+
+
+if request.method == "POST":
+    pass
+
+app.config["BAR"] = "BAZ"
+app.register_blueprint(blueprint=object())


Test patch:
Repository: semgrep/semgrep-rules

Issue:
View function arguments should match the URL parameters in the route decorator.



Discussion:
This is impossible to do in sgrep at the moment because we need to run string manipulation on metavariables! 

Resolving patch:
diff --git a/python/flask/different-route-names.py b/python/flask/different-route-names.py
new file mode 100644
--- /dev/null
+++ b/python/flask/different-route-names.py
@@ -0,0 +1,16 @@
+# ok
+@app.route("/api/snippet/<snippit_id>")
+@cache.cached(timeout=None)  # cache until restart or manual invalidation b/c immutable
+def get_snippet(snippit_id):
+    db_id = hashids.decode(snippit_id)
+
+# ruleid:flask-view-func-match-route-params
+@app.route("/api/snippet/<snippit_id>/<bar>")
+def get_snippet(snippit, baz):
+    db_id = hashids.decode(snippit_id)
+
+# ruleid:flask-view-func-match-route-params
+@app.route("/api/snippet")
+@cache.cached(timeout=None)  # cache until restart or manual invalidation b/c immutable
+def get_snippet(snippit_id):
+    db_id = hashids.decode(snippit_id)
\ No newline at end of file


Test patch:
Repository: semgrep/semgrep-rules

Issue:
User input format string into flask.make_response
This issue provides a great example

https://github.com/kylewm/silo.pub/issues/57

https://github.com/kylewm/silo.pub/blob/46aece85f8918f56ed75f1e11b544c10f70a17fc/silopub/micropub.py#L107


Discussion:

Resolving patch:
diff --git a/python/flask/unsanitized_input.py b/python/flask/unsanitized_input.py
new file mode 100644
--- /dev/null
+++ b/python/flask/unsanitized_input.py
@@ -0,0 +1,26 @@
+from flask import make_response
+
+def test1():
+    # ruleid: response_contains_unsanitized_input
+    x = request.args.get("x")
+    return make_response("found {}".format(x))
+
+
+def test1():
+    # ruleid: response_contains_unsanitized_input
+    x = request.args.get("x")
+    y = make_response("found {}".format(x))
+    return y
+
+
+def test2():
+    # ok
+    x = request.args.get("x")
+    y = some_safe_operation_on(x)
+    return make_response("found {}".format(y))
+
+
+def test3():
+    # ruleid: response_contains_unsanitized_input
+    x = request.args.get("x")
+    return make_response(f"found {x}")


Test patch:
Repository: scalableminds/webknossos-libs

Issue:
Tiled cubing doesn't convert last slice

Tiled cubing doesn't convert last slice



Discussion:

Resolving patch:
diff --git a/wkcuber/utils.py b/wkcuber/utils.py
--- a/wkcuber/utils.py
+++ b/wkcuber/utils.py
@@ -63,9 +63,10 @@ def get_chunks(arr, chunk_size):
         yield arr[i : i + chunk_size]
 
 
+# min_z and max_z are both inclusive
 def get_regular_chunks(min_z, max_z, chunk_size):
     i = floor(min_z / chunk_size) * chunk_size
-    while i < ceil(max_z / chunk_size) * chunk_size:
+    while i < ceil((max_z + 1) / chunk_size) * chunk_size:
         yield range(i, i + chunk_size)
         i += chunk_size
 


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -18,3 +18,12 @@ def test_get_regular_chunks():
     assert len(target) == 6
     assert list(target[0]) == list(range(0, 8))
     assert list(target[-1]) == list(range(40, 48))
+
+
+def test_get_regular_chunks_max_inclusive():
+    target = list(get_regular_chunks(4, 44, 1))
+
+    assert len(target) == 41
+    assert list(target[0]) == list(range(4, 5))
+    # The last chunk should include 44
+    assert list(target[-1]) == list(range(44, 45))
Repository: scalableminds/webknossos-libs

Issue:
elementClass is missing in datasource-properties.json when no mag1 resolution is available



Discussion:

Resolving patch:
diff --git a/wkcuber/metadata.py b/wkcuber/metadata.py
--- a/wkcuber/metadata.py
+++ b/wkcuber/metadata.py
@@ -151,8 +151,6 @@ def detect_standard_layer(dataset_path, layer_name, exact_bounding_box=None):
     else:
         bbox = exact_bounding_box
 
-    dtype = detect_dtype(dataset_path, layer_name)
-
     mags = list(detect_resolutions(dataset_path, layer_name))
     mags = sorted(mags)
     resolutions = [
@@ -163,6 +161,9 @@ def detect_standard_layer(dataset_path, layer_name, exact_bounding_box=None):
         for mag in mags
     ]
 
+    assert len(mags) > 0, "No resolutions found"
+    dtype = detect_dtype(dataset_path, layer_name, mags[0])
+
     return {
         "dataFormat": "wkw",
         "name": layer_name,


Test patch:
Repository: scalableminds/webknossos-libs

Issue:
Scale should not be required for metadata refresh
`--scale` should not be required for `wkcuber.metadata --refresh`


Discussion:

Resolving patch:
diff --git a/wkcuber/downsampling.py b/wkcuber/downsampling.py
--- a/wkcuber/downsampling.py
+++ b/wkcuber/downsampling.py
@@ -8,7 +8,7 @@
 from itertools import product
 from enum import Enum
 from .mag import Mag
-from .metadata import read_datasource_properties
+from .metadata import read_datasource_properties, refresh_metadata
 
 from .utils import (
     add_verbose_flag,
@@ -649,3 +649,5 @@ def detect_larger_and_smaller_dimension(scale):
             not args.no_compress,
             args,
         )
+
+    refresh_metadata(args.path)
diff --git a/wkcuber/metadata.py b/wkcuber/metadata.py
--- a/wkcuber/metadata.py
+++ b/wkcuber/metadata.py
@@ -36,7 +36,7 @@ def create_parser():
     )
     group.add_argument("--max_id", help="set max id of segmentation.", default=0)
 
-    add_scale_flag(parser)
+    add_scale_flag(parser, required=False)
     add_verbose_flag(parser)
 
     return parser
@@ -290,6 +290,9 @@ def detect_layers(dataset_path, max_id, compute_max_id, exact_bounding_box=None)
     setup_logging(args)
 
     if not args.refresh:
+        assert (
+            args.scale is not None
+        ), "The scale has to be specified when creating metadata for a dataset."
         assert (
             args.name is not None
         ), "Please provide a name via --name to create meta data."
diff --git a/wkcuber/utils.py b/wkcuber/utils.py
--- a/wkcuber/utils.py
+++ b/wkcuber/utils.py
@@ -61,12 +61,12 @@ def add_verbose_flag(parser):
     parser.set_defaults(verbose=True)
 
 
-def add_scale_flag(parser):
+def add_scale_flag(parser, required=True):
     parser.add_argument(
         "--scale",
         "-s",
         help="Scale of the dataset (e.g. 11.2,11.2,25). This is the size of one voxel in nm.",
-        required=True,
+        required=required,
     )
 
 


Test patch:
Repository: scalableminds/webknossos-libs

Issue:
Cubing a dataset with scale 320, 320, 200 will create a 1-2-2 mag
Instead, 2-2-1 should be created probably.
Cubing a dataset with scale 320, 320, 200 will create a 1-2-2 mag
Instead, 2-2-1 should be created probably.


Discussion:

Resolving patch:
diff --git a/wkcuber/downsampling.py b/wkcuber/downsampling.py
--- a/wkcuber/downsampling.py
+++ b/wkcuber/downsampling.py
@@ -632,12 +632,13 @@ def get_next_anisotropic_mag(mag, scale):
     max_index, min_index = detect_larger_and_smaller_dimension(scale)
     mag_array = mag.to_array()
     scale_increase = [1, 1, 1]
+
     if (
         mag_array[min_index] * scale[min_index]
         < mag_array[max_index] * scale[max_index]
     ):
         for i in range(len(scale_increase)):
-            scale_increase[i] = 1 if i == max_index else 2
+            scale_increase[i] = 1 if scale[i] == scale[max_index] else 2
     else:
         scale_increase = [2, 2, 2]
     return Mag(


Test patch:
diff --git a/tests/test_downsampling.py b/tests/test_downsampling.py
--- a/tests/test_downsampling.py
+++ b/tests/test_downsampling.py
@@ -207,14 +207,15 @@ def test_anisotropic_mag_calculation():
         [(10.5, 24, 10.5), Mag((2, 1, 2)), Mag((4, 1, 4))],
         [(24, 10.5, 10.5), Mag((1, 2, 2)), Mag((1, 4, 4))],
         [(10.5, 10.5, 10.5), Mag(2), Mag(4)],
+        [(320, 320, 200), Mag(1), Mag((1, 1, 2))],
+        [(320, 320, 200), Mag((1, 1, 2)), Mag((2, 2, 4))],
     ]
+
     for i in range(len(mag_tests)):
-        assert mag_tests[i][2] == get_next_anisotropic_mag(
-            mag_tests[i][1], mag_tests[i][0]
-        ), (
+        next_mag = get_next_anisotropic_mag(mag_tests[i][1], mag_tests[i][0])
+        assert mag_tests[i][2] == next_mag, (
             "The next anisotropic"
-            " Magnification of {} with "
-            "the size {} should be {}".format(
-                mag_tests[i][1], mag_tests[i][0], mag_tests[i][2]
-            )
+            f" Magnification of {mag_tests[i][1]} with "
+            f"the size {mag_tests[i][0]} should be {mag_tests[i][2]} "
+            f"and not {next_mag}"
         )
Repository: scalableminds/webknossos-libs

Issue:
NIFTI Division by zero error
When converting a nifti file only consisting of zeros, the to_target_datatype function divides by 0.


Discussion:

Resolving patch:
diff --git a/wkcuber/convert_nifti.py b/wkcuber/convert_nifti.py
--- a/wkcuber/convert_nifti.py
+++ b/wkcuber/convert_nifti.py
@@ -1,21 +1,22 @@
-import time
 import logging
-import numpy as np
+import time
 from argparse import ArgumentParser
 from pathlib import Path
+
 import nibabel as nib
+import numpy as np
 
-from wkcuber.api.Dataset import WKDataset, TiffDataset
-from .utils import (
-    add_verbose_flag,
-    setup_logging,
+from wkcuber.api.Dataset import TiffDataset, WKDataset
+from wkcuber.utils import (
+    DEFAULT_WKW_FILE_LEN,
+    DEFAULT_WKW_VOXELS_PER_BLOCK,
     add_scale_flag,
+    add_verbose_flag,
     pad_or_crop_to_size_and_topleft,
     parse_bounding_box,
+    setup_logging,
 )
 
-from wkcuber.utils import DEFAULT_WKW_FILE_LEN, DEFAULT_WKW_VOXELS_PER_BLOCK
-
 
 def create_parser():
     parser = ArgumentParser()
@@ -102,6 +103,10 @@ def to_target_datatype(
     else:
         factor = np.iinfo(data.dtype).max / np.iinfo(target_dtype).max
 
+    if data.max() == 0:
+        logging.warning("Not rescaling data since maximum is 0")
+        factor = 1
+
     return (data / factor).astype(np.dtype(target_dtype))
 
 


Test patch:
Repository: scalableminds/webknossos-libs

Issue:
Convenience for wkcuber.api
To open/create a dataset with the cool new high-level API the following code is required:

```python
from wkcuber.api.Dataset import WKDataset
from pathlib import Path

ds1 = WKDataset.create(Path("path") / "to" / "dataset1", scale=(128,128,128))
ds2 = WKDataset.open(Path("path") / "to" / "dataset2")

```

For one-off scripts, I think that could be a bit more convenient, if we had an API like this

```python
from wkcuber import WKDataset

ds1 = WKDataset.create("path/to/dataset1", scale=(128, 128, 128))
ds2 = WKDataset.open("path/to/dataset2")
```

Any thoughts? @rschwanhold @jstriebel @philippotto 



Discussion:
Context: I have been using the API a couple of times now to convert datasets for users, where the automatic conversion didn't succeed yet.
Sounds fair for me. For public-facing apis, that are not only used in vx or other internal code, allowing `Union[Path, str]` makes sense.
Yes, fully agree :+1: 
Is this limited to the Dataset API or are there more cases like this?
I think `WKDataset` is the primary entry point for the API, so there would be the biggest benefit. I don't think we need to expose the whole API in this manner. 
My initial request also contained re-exporting `WKDataset` under `wkcuber`:

```python
from wkcuber import WKDataset
```

Resolving patch:
diff --git a/wkcuber/__init__.py b/wkcuber/__init__.py
--- a/wkcuber/__init__.py
+++ b/wkcuber/__init__.py
@@ -1,4 +1,6 @@
+from .api.Dataset import WKDataset
 from .cubing import cubing
 from .downsampling import downsample_mags
 from .compress import compress_mag
+from .mag import Mag
 from .metadata import write_webknossos_metadata


Test patch:
diff --git a/tests/test_reexport.py b/tests/test_reexport.py
new file mode 100644
--- /dev/null
+++ b/tests/test_reexport.py
@@ -0,0 +1,8 @@
+from wkcuber import Mag, WKDataset
+from wkcuber.mag import Mag as _Mag
+from wkcuber.api.Dataset import WKDataset as _WKDataset
+
+
+def test_reexport_classes() -> None:
+    assert Mag == _Mag, "Mag exports should be the same class"
+    assert WKDataset == _WKDataset, "WKDataset exports should be the same class"
Repository: scalableminds/webknossos-libs

Issue:
Usage examples in README.md
Some of the usage examples in README.md are not compatible with the latest version of this package. For example, `python -m wkcuber` does not take `--layer_name` as an option as of v0.5.0.


Discussion:

Resolving patch:
diff --git a/wkcuber/converter.py b/wkcuber/converter.py
--- a/wkcuber/converter.py
+++ b/wkcuber/converter.py
@@ -76,6 +76,12 @@ def get_source_files(
     return source_files
 
 
+def all_files_of_same_type(input_files: List[str]) -> bool:
+    _, ext = path.splitext(input_files[0])
+
+    return all(map(lambda p: path.splitext(p)[1] == ext, input_files))
+
+
 class Converter:
     def __init__(self) -> None:
         self.source_files: List[str] = []
@@ -329,12 +335,6 @@ def accepts_input(self, source_path: str) -> bool:
         if len(source_files) == 0:
             return False
 
-        _, ext = path.splitext(source_files[0])
-
-        assert all(
-            map(lambda p: path.splitext(p)[1] == ext, source_files)
-        ), "Not all image files are of the same type"
-
         self.source_files = list(
             map(lambda p: cast(str, path.normpath(p)), source_files)
         )
@@ -377,7 +377,17 @@ def convert_input(self, args: Namespace) -> bool:
 
         bounding_box = None
         view_configuration = dict()
+        converted_layers = 0
         for layer_path, layer_name in layer_path_to_name.items():
+            if not all_files_of_same_type(
+                get_source_files(layer_path, image_reader.readers.keys(), True)
+            ):
+                logger.info(
+                    f"Not converting {layer_name} because not all image files are of the same type"
+                )
+                continue
+
+            converted_layers += 1
             channel_count, dtype = get_channel_count_and_dtype(Path(layer_path))
             if channel_count > 1 and not (channel_count == 3 and dtype == "uint8"):
                 for i in range(channel_count):
@@ -419,6 +429,8 @@ def convert_input(self, args: Namespace) -> bool:
                     executor_args,
                 )
 
+        assert converted_layers > 0, "No layer could be converted!"
+
         write_webknossos_metadata(
             args.target_path,
             args.name,


Test patch:
Repository: scalableminds/webknossos-libs

Issue:
BoundingBox and BoundingBoxNamedTuple should have bidirectional conversion
A lot of voxelytics code does nothing but converting these, which could be DRYer by moving the conversion into the API


Discussion:

Resolving patch:
diff --git a/wkcuber/api/bounding_box.py b/wkcuber/api/bounding_box.py
--- a/wkcuber/api/bounding_box.py
+++ b/wkcuber/api/bounding_box.py
@@ -143,6 +143,12 @@ def as_csv(self) -> str:
 
         return ",".join(map(str, self.as_tuple6()))
 
+    def as_named_tuple(self) -> BoundingBoxNamedTuple:
+        return BoundingBoxNamedTuple(
+            topleft=cast(Tuple[int, int, int], tuple(self.topleft)),
+            size=cast(Tuple[int, int, int], tuple(self.size)),
+        )
+
     def __repr__(self) -> str:
 
         return "BoundingBox(topleft={}, size={})".format(


Test patch:
Repository: scalableminds/webknossos-libs

Issue:
Fix regexes that assume unix-paths
Some regexes use `/`, which doesn't work under Windows. We should go through the code and replace these regexes with path walks or fix make the regexes universal.


Discussion:

Resolving patch:
diff --git a/wkcuber/knossos.py b/wkcuber/knossos.py
--- a/wkcuber/knossos.py
+++ b/wkcuber/knossos.py
@@ -11,7 +11,9 @@
 CUBE_EDGE_LEN = 128
 CUBE_SIZE = CUBE_EDGE_LEN ** 3
 CUBE_SHAPE = (CUBE_EDGE_LEN,) * 3
-CUBE_REGEX = re.compile(r"x(\d+)/y(\d+)/z(\d+)/(.*\.raw)$")
+CUBE_REGEX = re.compile(
+    fr"x(\d+){os.path.sep}y(\d+){os.path.sep}z(\d+){os.path.sep}(.*\.raw)$"
+)
 
 
 class KnossosDataset:
diff --git a/wkcuber/metadata.py b/wkcuber/metadata.py
--- a/wkcuber/metadata.py
+++ b/wkcuber/metadata.py
@@ -1,4 +1,5 @@
 import json
+import os
 import re
 import wkw
 import logging
@@ -235,7 +236,9 @@ def list_files(layer_path: str) -> Iterable[str]:
         return iglob(path.join(layer_path, "*", "*", "*.wkw"), recursive=True)
 
     def parse_cube_file_name(filename: str) -> Tuple[int, int, int]:
-        CUBE_REGEX = re.compile(r"z(\d+)/y(\d+)/x(\d+)(\.wkw)$")
+        CUBE_REGEX = re.compile(
+            fr"z(\d+){os.path.sep}y(\d+){os.path.sep}x(\d+)(\.wkw)$"
+        )
         m = CUBE_REGEX.search(filename)
         if m is not None:
             return int(m.group(3)), int(m.group(2)), int(m.group(1))
diff --git a/wkcuber/utils.py b/wkcuber/utils.py
--- a/wkcuber/utils.py
+++ b/wkcuber/utils.py
@@ -49,7 +49,7 @@
 BLOCK_LEN = 32
 DEFAULT_WKW_FILE_LEN = 32
 DEFAULT_WKW_VOXELS_PER_BLOCK = 32
-CUBE_REGEX = re.compile(r"z(\d+)/y(\d+)/x(\d+)(\.wkw)$")
+CUBE_REGEX = re.compile(fr"z(\d+){os.path.sep}y(\d+){os.path.sep}x(\d+)(\.wkw)$")
 
 logger = getLogger(__name__)
 


Test patch: