internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: adamchainz/django-perf-rec

Issue:
Doesn't capture cache.get_or_set
I would expect that `cache.get_or_set` either be recorded as a `get` and (optionally) a `set`, or perhaps as its own entry. Instead it seems to be ignored completely:
``` python
class TestCache(TestCase):
    def test_get(self):
        with django_perf_rec.record():
            django_cache.get('foo')

    def test_set(self):
        with django_perf_rec.record():
            django_cache.set('foo', 42)

    def test_get_or_set(self):
        with django_perf_rec.record():
            django_cache.get_or_set('foo', 42)
```
This results in
``` yaml
TestCache.test_get:
- cache|get: foo
TestCache.test_get_or_set: []
TestCache.test_set:
- cache|set: foo
```
It looks like this is a result of some code which aims to identify "internal" usages of the cache, though I'm not really sure.


Discussion:

Resolving patch:
diff --git a/src/django_perf_rec/cache.py b/src/django_perf_rec/cache.py
--- a/src/django_perf_rec/cache.py
+++ b/src/django_perf_rec/cache.py
@@ -120,6 +120,7 @@ def __exit__(self, _, __, ___):
         "delete_many",
         "get",
         "get_many",
+        "get_or_set",
         "incr",
         "set",
         "set_many",


Test patch:
diff --git a/tests/test_api.perf.yml b/tests/test_api.perf.yml
--- a/tests/test_api.perf.yml
+++ b/tests/test_api.perf.yml
@@ -1,8 +1,10 @@
 RecordTests.test_delete_on_cascade_called_twice:
-- db: 'DELETE FROM "testapp_book" WHERE "testapp_book"."author_id" IN (#)'
-- db: 'DELETE FROM "testapp_award" WHERE "testapp_award"."author_id" IN (#)'
-- db: 'DELETE FROM "testapp_contract_author" WHERE "testapp_contract_author"."author_id" IN (#)'
-- db: 'DELETE FROM "testapp_author" WHERE "testapp_author"."id" IN (#)'
+- db: DELETE FROM "testapp_book" WHERE "testapp_book"."author_id" IN (#)
+- db: DELETE FROM "testapp_award" WHERE "testapp_award"."author_id" IN (#)
+- db: DELETE FROM "testapp_contract_author" WHERE "testapp_contract_author"."author_id" IN (#)
+- db: DELETE FROM "testapp_author" WHERE "testapp_author"."id" IN (#)
+RecordTests.test_get_or_set:
+- cache|get_or_set: foo
 RecordTests.test_multiple_cache_ops:
 - cache|set: foo
 - cache|second|get_many:
diff --git a/tests/test_api.py b/tests/test_api.py
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -69,6 +69,10 @@ def test_single_cache_op(self):
         with record():
             caches["default"].get("foo")
 
+    def test_get_or_set(self):
+        with record():
+            caches["default"].get_or_set("foo", 42)
+
     def test_single_cache_op_with_traceback(self):
         with pretend_not_under_pytest():
             with pytest.raises(AssertionError) as excinfo:
Repository: adamchainz/django-perf-rec

Issue:
IndexError from unconditional UPDATE queries
I think there's a bug in `sql_recursively_simplify` such that it cannot handle `UPDATE` queries that do not contain a `WHERE` clause.

For example:
``` python
with django_perf_rec.record():
    Author.objects.update(name=None)
```
The traceback I get (limited to the frames in `django-perf-rec`) is:
```
  File ".../site-packages/django_perf_rec/db.py", line 62, in inner
    alias=alias, sql=sql_fingerprint(sql, hide_columns=hide_columns)
  File ".../site-packages/django_perf_rec/sql.py", line 20, in sql_fingerprint
    sql_recursively_simplify(parsed_query, hide_columns=hide_columns)
  File ".../site-packages/django_perf_rec/sql.py", line 80, in sql_recursively_simplify
    ][0]
IndexError: list index out of range
```


Discussion:

Resolving patch:
diff --git a/src/django_perf_rec/sql.py b/src/django_perf_rec/sql.py
--- a/src/django_perf_rec/sql.py
+++ b/src/django_perf_rec/sql.py
@@ -73,13 +73,18 @@ def sql_recursively_simplify(node, hide_columns=True):
     # Erase which fields are being updated in an UPDATE
     if node.tokens[0].value == "UPDATE":
         i_set = [i for (i, t) in enumerate(node.tokens) if t.value == "SET"][0]
-        i_where = [
+        i_wheres = [
             i
             for (i, t) in enumerate(node.tokens)
             if t.is_group and t.tokens[0].value == "WHERE"
-        ][0]
+        ]
+        if i_wheres:
+            i_where = i_wheres[0]
+            end = node.tokens[i_where:]
+        else:
+            end = []
         middle = [Token(tokens.Punctuation, " ... ")]
-        node.tokens = node.tokens[: i_set + 1] + middle + node.tokens[i_where:]
+        node.tokens = node.tokens[: i_set + 1] + middle + end
 
     # Ensure IN clauses with simple value in always simplify to "..."
     if node.tokens[0].value == "WHERE":


Test patch:
diff --git a/tests/test_sql.py b/tests/test_sql.py
--- a/tests/test_sql.py
+++ b/tests/test_sql.py
@@ -184,6 +184,12 @@ def test_update():
     )
 
 
+def test_update_no_where():
+    assert (
+        sql_fingerprint("UPDATE `table` SET `foo` = 'bar'") == "UPDATE `table` SET ..."
+    )
+
+
 def test_declare_cursor():
     assert (
         sql_fingerprint(
Repository: adamchainz/django-perf-rec

Issue:
overwrite MODE can result in invalid snapshot files
### Python Version

3.9

### Django Version

3.2

### Package Version

4.20.0

### Description

If the new snapshot data are smaller than the previous ones, then `overwrite` mode can result in the file contents containing trailing data left over from the previous snapshots. In turn that data can end up being invalid YAML which then causes issues when the file is next read.

It looks like `set_and_save` needs to call [`fp.truncate()`](https://docs.python.org/3/library/io.html#io.IOBase.truncate) after `yaml.safe_dump`.

If you agree with this approach I'd be happy to put together a PR.


Discussion:

Resolving patch:
diff --git a/src/django_perf_rec/yaml.py b/src/django_perf_rec/yaml.py
--- a/src/django_perf_rec/yaml.py
+++ b/src/django_perf_rec/yaml.py
@@ -76,3 +76,4 @@ def set_and_save(self, key: str, value: PerformanceRecord) -> None:
             yaml.safe_dump(
                 data, fp, default_flow_style=False, allow_unicode=True, width=10000
             )
+            fp.truncate()


Test patch:
diff --git a/tests/test_api.py b/tests/test_api.py
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -3,6 +3,7 @@
 import os
 
 import pytest
+import yaml
 from django.core.cache import caches
 from django.db.models import F, Q
 from django.db.models.functions import Upper
@@ -243,19 +244,19 @@ def test_mode_overwrite(self):
 
             with record(path="perf_files/api/", record_name="test_mode_overwrite"):
                 caches["default"].get("foo")
+                caches["default"].get("bar")
 
             full_path = os.path.join(FILE_DIR, "perf_files", "api", "test_api.perf.yml")
             assert os.path.exists(full_path)
 
             with record(path="perf_files/api/", record_name="test_mode_overwrite"):
-                caches["default"].get("bar")
+                caches["default"].get("baz")
 
             full_path = os.path.join(FILE_DIR, "perf_files", "api", "test_api.perf.yml")
             with open(full_path) as f:
-                text = f.read()
+                data = yaml.safe_load(f.read())
 
-            assert "bar" in text
-            assert "foo" not in text
+            assert data == {"test_mode_overwrite": [{"cache|get": "baz"}]}
 
     def test_delete_on_cascade_called_twice(self):
         arthur = Author.objects.create(name="Arthur", age=42)
Repository: adamchainz/django-perf-rec

Issue:
Current django_perf_rec implementation is not compatible with the usage of named "key" parameter to set_cache function
### Python Version

3.9.16

### Django Version

3.2.16

### Package Version

4.21.0

### Description

The django.core.cache definition of set_cache allows the naming of all parameters.
`cache.set(key="key", value="value")` is therefore acceptable

See [https://docs.djangoproject.com/en/3.2/topics/cache/#django.core.caches.cache.set](https://docs.djangoproject.com/en/3.2/topics/cache/#django.core.caches.cache.set)

However, django_perf_rec's current implementation is incompatible with this approach as it directly accesses `args[0]` and considers it as the "key" parameter.

In `cache.py`, the line which I believe is faulty is the following: `key_or_keys = args[0]`
This should be replaced with a more robust approach: `key_or_keys = args[0] if args else kwargs['key']` or, better yet(?) declare `key` as a named parameter directly in the parameters of the function overload.


Discussion:

Resolving patch:
diff --git a/src/django_perf_rec/cache.py b/src/django_perf_rec/cache.py
--- a/src/django_perf_rec/cache.py
+++ b/src/django_perf_rec/cache.py
@@ -109,7 +109,12 @@ def inner(self: Any, *args: Any, **kwargs: Any) -> Any:
                     del frame
 
                 if not is_internal_call:
-                    key_or_keys = args[0]
+                    if args:
+                        key_or_keys = args[0]
+                    elif "key" in kwargs:
+                        key_or_keys = kwargs["key"]
+                    else:
+                        key_or_keys = kwargs["keys"]
                     callback(
                         CacheOp(
                             alias=alias,


Test patch:
diff --git a/tests/test_cache.py b/tests/test_cache.py
--- a/tests/test_cache.py
+++ b/tests/test_cache.py
@@ -132,11 +132,15 @@ def test_records_all(self, stack_summary):
         callback = mock.Mock()
         with AllCacheRecorder(callback):
             caches["default"].get("foo")
+            caches["default"].get(key="foo")
+            caches["default"].get_many(keys=["foo"])
             caches["second"].set("bar", "baz")
             caches["default"].delete_many(["foo"])
 
         assert callback.mock_calls == [
             mock.call(CacheOp("default", "get", "foo", stack_summary)),
+            mock.call(CacheOp("default", "get", "foo", stack_summary)),
+            mock.call(CacheOp("default", "get_many", ["foo"], stack_summary)),
             mock.call(CacheOp("second", "set", "bar", stack_summary)),
             mock.call(CacheOp("default", "delete_many", ["foo"], stack_summary)),
         ]
Repository: ultralytics/yolov5

Issue:
Distributed training in cluster not correct.
The model can be trained correctly on single machine multi gpus.
However, It can't be trained on multi machines multi gpus.


Discussion:

Resolving patch:
diff --git a/train.py b/train.py
--- a/train.py
+++ b/train.py
@@ -151,6 +151,7 @@ def train(hyp):
                                 world_size=1,  # number of nodes
                                 rank=0)  # node rank
         model = torch.nn.parallel.DistributedDataParallel(model)
+        # pip install torch==1.4.0+cu100 torchvision==0.5.0+cu100 -f https://download.pytorch.org/whl/torch_stable.html
 
     # Dataset
     dataset = LoadImagesAndLabels(train_path, imgsz, batch_size,


Test patch:
Repository: ultralytics/yolov5

Issue:
TypeError: can't pickle torch.distributed.ProcessGroupNCCL objects
Hi，
I meet a problem:

Traceback (most recent call last):
  File "train.py", line 394, in <module>
    train(hyp)
  File "train.py", line 331, in train
    torch.save(ckpt, last)
  File "/home/yy/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 328, in save
    _legacy_save(obj, opened_file, pickle_module, pickle_protocol)
  File "/home/yy/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 401, in _legacy_save
    pickler.dump(obj)
**TypeError: can't pickle torch.distributed.ProcessGroupNCCL objects**

Thanks!

environment:
ubuntu 16.04
GPU 2080Ti *4
pytorch 1.4.0



Discussion:

Resolving patch:
diff --git a/utils/torch_utils.py b/utils/torch_utils.py
--- a/utils/torch_utils.py
+++ b/utils/torch_utils.py
@@ -201,5 +201,5 @@ def update(self, model):
     def update_attr(self, model):
         # Update EMA attributes
         for k, v in model.__dict__.items():
-            if not k.startswith('_') and k != 'module':
+            if not k.startswith('_') and k not in ["process_group", "reducer"]:
                 setattr(self.ema, k, v)


Test patch:
Repository: ultralytics/yolov5

Issue:
utils.general.init_seeds() raises RecursionError
## 🐛 Bug
`utils.general.init_seeds()` raises `RecursionError`

## To Reproduce

Input:
```
from utils.general import init_seeds

init_seeds()
```

Output:
```
RecursionError                            Traceback (most recent call last)
<ipython-input-2-f6caabb30a03> in <module>
----> 1 init_seeds()

~/git/yolov5/utils/general.py in init_seeds(seed)
     57     random.seed(seed)
     58     np.random.seed(seed)
---> 59     init_seeds(seed=seed)
     60 
     61 

... last 1 frames repeated, from the frame below ...

~/git/yolov5/utils/general.py in init_seeds(seed)
     57     random.seed(seed)
     58     np.random.seed(seed)
---> 59     init_seeds(seed=seed)
     60 
     61 

RecursionError: maximum recursion depth exceeded while calling a Python object
```

## Environment
 - OS: Ubuntu 16.04
 - GPU: 1080 Ti


Discussion:

Resolving patch:
diff --git a/utils/general.py b/utils/general.py
--- a/utils/general.py
+++ b/utils/general.py
@@ -23,7 +23,8 @@
 from scipy.signal import butter, filtfilt
 from tqdm import tqdm
 
-from utils.torch_utils import init_seeds, is_parallel
+from utils.torch_utils import init_seeds as init_torch_seeds
+from utils.torch_utils import is_parallel
 
 # Set printoptions
 torch.set_printoptions(linewidth=320, precision=5, profile='long')
@@ -55,7 +56,7 @@ def set_logging(rank=-1):
 def init_seeds(seed=0):
     random.seed(seed)
     np.random.seed(seed)
-    init_seeds(seed=seed)
+    init_torch_seeds(seed=seed)
 
 
 def get_latest_run(search_dir='./runs'):


Test patch:
Repository: ultralytics/yolov5

Issue:
Grid indices overflow bug fix
This PR addresses bug report #1312. In extremely rare cases, when training with mixed precision the grid index values in compute_targets() may experience overflow. This solves this problem by clamping the indices to their limits.

## 🛠️ PR Summary

<sub>Made with ❤️ by [Ultralytics Actions](https://github.com/ultralytics/actions)<sub>

### 🌟 Summary
Improvements to grid clamping in build targets and dynamic image resizing in plot images.

### 📊 Key Changes
- Imported `math` module is moved to organize imports in `general.py`.
- Updated grid clamping logic in the `build_targets` function to prevent out-of-bound grid indices.
- Modified image resizing behavior in `plot_images` to cap the image size for enhancing visualization and reducing memory usage.

### 🎯 Purpose & Impact
- 🎲 **Grid Clamping**: To ensure generated grid indices do not exceed model's grid dimensions, enhancing model stability.
- 🔍 **Dynamic Image Resizing**: To dynamically adjust the image size during plotting to optimize for visibility while keeping file sizes manageable.
- 🖼️ **Visualization Improvement**: Better plotting of images without degradation or excessive resizing, resulting in improved user experience when reviewing output images.


Discussion:
Also observed in YOLOv3 https://github.com/ultralytics/yolov3/issues/1551

Resolving patch:
diff --git a/utils/general.py b/utils/general.py
--- a/utils/general.py
+++ b/utils/general.py
@@ -589,7 +589,7 @@ def build_targets(p, targets, model):
 
         # Append
         a = t[:, 6].long()  # anchor indices
-        indices.append((b, a, gj.clamp_(0, gain[3]), gi.clamp_(0, gain[2])))  # image, anchor, grid indices
+        indices.append((b, a, gj.clamp_(0, gain[3] - 1), gi.clamp_(0, gain[2] - 1)))  # image, anchor, grid indices
         tbox.append(torch.cat((gxy - gij, gwh), 1))  # box
         anch.append(anchors[a])  # anchors
         tcls.append(c)  # class


Test patch:
Repository: ultralytics/yolov5

Issue:
Caching image path
I have 2 files train.txt and valid.txt. The train images and valid images are in the same directory "/content/train"

```
# train.txt file
/content/train/1.jpg
/content/train/2.jpg
# val.txt file
/content/train/3.jpg
/content/train/4.jpg
```
When we train a model, the train and val caching file is the same name `train.cache`
```
...
Scanning images: 100% 14915/14915 [6:48:26<00:00,  1.64s/it]
Scanning labels /content/train.cache (14915 found, 0 missing, 0 empty, 0 duplicate, for 14915 images): 14915it [00:00, 15900.80it/s]
Scanning images: 100% 8657/8657 [02:26<00:00, 59.22it/s]
Scanning labels /content/train.cache (8657 found, 0 missing, 0 empty, 0 duplicate, for 8657 images): 8657it [00:00, 16335.68it/s]
...
```


Discussion:

Resolving patch:
diff --git a/utils/datasets.py b/utils/datasets.py
--- a/utils/datasets.py
+++ b/utils/datasets.py
@@ -372,7 +372,7 @@ def __init__(self, path, img_size=640, batch_size=16, augment=False, hyp=None, r
 
         # Check cache
         self.label_files = img2label_paths(self.img_files)  # labels
-        cache_path = Path(self.label_files[0]).parent.with_suffix('.cache')  # cached labels
+        cache_path = (p if p.is_file() else Path(self.label_files[0]).parent).with_suffix('.cache')  # cached labels
         if cache_path.is_file():
             cache = torch.load(cache_path)  # load
             if cache['hash'] != get_hash(self.label_files + self.img_files) or 'results' not in cache:  # changed


Test patch:
Repository: ultralytics/yolov5

Issue:
DDP wandb utils not running check_file() on --data
## 🐛 Bug

@AyushExel I found a bit of a bug today on starting a DDP training with `--data coco.yaml`. All our main python files (train, test, detect) are set up so that files passed to the argparser are searched locally inside the yolov5 repo using `file = check_file(file)`. This allows for more concise commands like --cfg yolov5s.yaml rather than --cfg models/yolov5s.yaml, but it looks like opt.data got passed to the wandb utils before it had a chance to pass through check_file() first. Should be an easy fix, I think I can do this tomorrow.

<img width="1214" alt="Screenshot 2021-05-05 at 01 46 55" src="https://user-images.githubusercontent.com/26833433/117083255-94c12780-ad44-11eb-85c2-48bef8943717.png">

## To Reproduce (REQUIRED)

Start DDP training with `--data coco.yaml`
```
 python -m torch.distributed.launch --nproc_per_node 2 --master_port 1 train.py --batch 64 --cfg yolov5x.yaml --weights '' --data coco.yaml
```



Discussion:

Resolving patch:
diff --git a/utils/wandb_logging/wandb_utils.py b/utils/wandb_logging/wandb_utils.py
--- a/utils/wandb_logging/wandb_utils.py
+++ b/utils/wandb_logging/wandb_utils.py
@@ -9,7 +9,7 @@
 sys.path.append(str(Path(__file__).parent.parent.parent))  # add utils/ to path
 from utils.datasets import LoadImagesAndLabels
 from utils.datasets import img2label_paths
-from utils.general import colorstr, xywh2xyxy, check_dataset
+from utils.general import colorstr, xywh2xyxy, check_dataset, check_file
 
 try:
     import wandb
@@ -54,7 +54,7 @@ def check_wandb_resume(opt):
 
 
 def process_wandb_config_ddp_mode(opt):
-    with open(opt.data) as f:
+    with open(check_file(opt.data)) as f:
         data_dict = yaml.safe_load(f)  # data dict
     train_dir, val_dir = None, None
     if isinstance(data_dict['train'], str) and data_dict['train'].startswith(WANDB_ARTIFACT_PREFIX):
@@ -115,7 +115,7 @@ def __init__(self, opt, name, run_id, data_dict, job_type='Training'):
     def check_and_upload_dataset(self, opt):
         assert wandb, 'Install wandb to upload dataset'
         check_dataset(self.data_dict)
-        config_path = self.log_dataset_artifact(opt.data,
+        config_path = self.log_dataset_artifact(check_file(opt.data),
                                                 opt.single_cls,
                                                 'YOLOv5' if opt.project == 'runs/train' else Path(opt.project).stem)
         print("Created dataset config file ", config_path)


Test patch:
Repository: ultralytics/yolov5

Issue:
model ensembling isn't working
## 🐛 Bug
When I detect some image by using ensembling, it doesn't work.

## To Reproduce (REQUIRED)
Input:
```
!git clone https://github.com/ultralytics/yolov5  # clone repo
%cd yolov5
%pip install -qr requirements.txt  # install dependencies

import torch
from IPython.display import Image, clear_output  # to display images
import urllib

clear_output()
print(f"Setup complete. Using torch {torch.__version__} ({torch.cuda.get_device_properties(0).name if torch.cuda.is_available() else 'CPU'})")

def download_file(url, dst_path):
    try:
        with urllib.request.urlopen(url) as web_file:
            data = web_file.read()
            with open(dst_path, mode='wb') as local_file:
                local_file.write(data)
    except urllib.error.URLError as e:
        print(e)

download_file('https://user-images.githubusercontent.com/26833433/124489091-ea4f9a00-ddb0-11eb-8ef1-d6f335c97f6f.jpg', "zidane.jpg")

!python detect.py --weights yolov5x.pt yolov5s.pt --img 640 --conf 0.25 --source data/images/
Image(filename='zidane.jpg', width=600)
```

Output:
```
image 1/2 /content/yolov5/yolov5/yolov5/yolov5/data/images/bus.jpg: Traceback (most recent call last):
  File "detect.py", line 228, in <module>
    main(opt)
  File "detect.py", line 223, in main
    run(**vars(opt))
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "detect.py", line 106, in run
    visualize=increment_path(save_dir / Path(path).stem, mkdir=True) if visualize else False)[0]
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'visualize'
```
<img width="1093" alt="スクリーンショット 2021-07-12 1 31 10" src="https://user-images.githubusercontent.com/33506506/125202974-31f28c00-e2b1-11eb-8d50-ff518011c32e.png">


## Expected behavior
detect image with ensembling correctly.

## Environment
If applicable, add screenshots to help explain your problem.

google colab

https://colab.research.google.com/drive/1rXRjuFTiHdJwbxhSIY8EywwQMrg3zCbV?usp=sharing

 - OS: [e.g. Ubuntu]
 - GPU Tesla P100

## Additional context
I'm trying to fix it now. might be one day from now I will make pull request



Discussion:

Resolving patch:
diff --git a/models/experimental.py b/models/experimental.py
--- a/models/experimental.py
+++ b/models/experimental.py
@@ -100,10 +100,10 @@ class Ensemble(nn.ModuleList):
     def __init__(self):
         super(Ensemble, self).__init__()
 
-    def forward(self, x, augment=False):
+    def forward(self, x, augment=False, profile=False, visualize=False):
         y = []
         for module in self:
-            y.append(module(x, augment)[0])
+            y.append(module(x, augment, profile, visualize)[0])
         # y = torch.stack(y).max(0)[0]  # max ensemble
         # y = torch.stack(y).mean(0)  # mean ensemble
         y = torch.cat(y, 1)  # nms ensemble


Test patch: