internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ray-project/ray

Issue:
[java] Java usage is shown when importing from Python
### What is the problem?

*Ray version and other system information (Python version, TensorFlow version, OS):*
master

### Reproduction (REQUIRED)
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):
I get shown:
<img width="505" alt="Screen Shot 2020-09-11 at 12 50 54 PM" src="https://user-images.githubusercontent.com/4529381/92967051-701ee500-f42d-11ea-877c-c6ddc3fde1ef.png">

when calling `ray.init()`. The Python library should have no dependency on Java.


Discussion:
That was easy to find:

```
git diff origin/releases/0.8.7..origin/master python/ray/services.py
```

I betcha dollars to donuts its afa0216280409ff9806640658605e24b64d81ea7, specifically https://github.com/ray-project/ray/commit/afa0216280409ff9806640658605e24b64d81ea7#diff-54ac27010a06993004bec4677c7e583eR1329-R1332

what you really want is to do the equivalent of a `which java` instead of attempting to actually run `java` -- if someone has errantly created a java binary of their own, that's their own fault. Indeed, that dialog is MacOS "helpfully" doing exactly this

Resolving patch:
diff --git a/python/ray/services.py b/python/ray/services.py
--- a/python/ray/services.py
+++ b/python/ray/services.py
@@ -6,6 +6,7 @@
 import multiprocessing
 import os
 import random
+import shutil
 import signal
 import socket
 import subprocess
@@ -1314,15 +1315,8 @@ def start_raylet(redis_address,
     gcs_ip_address, gcs_port = redis_address.split(":")
 
     has_java_command = False
-    try:
-        java_proc = subprocess.run(
-            ["java", "-version"],
-            stdout=subprocess.PIPE,
-            stderr=subprocess.PIPE)
-        if java_proc.returncode == 0:
-            has_java_command = True
-    except OSError:
-        pass
+    if shutil.which("java") is not None:
+        has_java_command = True
 
     ray_java_installed = False
     try:


Test patch:
Repository: ray-project/ray

Issue:
[rllib] DDPG TWIN_Q LOSS ERROR
The TD3 loss according to OpenAI Spinning up is:
![L1](https://spinningup.openai.com/en/latest/_images/math/7d5c18f49a242cc3eec554f717fe4f3bfc119bab.svg)

![L2](https://spinningup.openai.com/en/latest/_images/math/cd73726a8a3845ade467aed57714912f868f6b36.svg)

They implement this as you would expect:
````
# MSE loss against Bellman backup
loss_q1 = ((q1 - backup)**2).mean()
loss_q2 = ((q2 - backup)**2).mean()
loss_q = loss_q1 + loss_q2
````

The rllib implementation is off and I ran into training issues when using it.

Tensorflow:
https://github.com/ray-project/ray/blob/39c598bab0929f007a49daa7262a7c9ee6492221/rllib/agents/ddpg/ddpg_tf_policy.py#L174-L184

Pytorch:
https://github.com/ray-project/ray/blob/5851e893eee491e569fe1421314172bd5b37d583/rllib/agents/ddpg/ddpg_torch_policy.py#L124-L134

The error is on line 177 and 127 respectively where the td_error and twin_td_error are added together and then squared a couple lines later. Removing that line would bring the implementation in line with the spinningup reference implementation. This worked for me to improve training on my problem. They also do not multiply by the constant (0.5) but I would not expect that to make much of a difference either way.



Discussion:
Interesting, do you want to push a fix. Also cc @michaelzhiluo if you could take a look here.

Resolving patch:
diff --git a/rllib/agents/ddpg/ddpg_tf_policy.py b/rllib/agents/ddpg/ddpg_tf_policy.py
--- a/rllib/agents/ddpg/ddpg_tf_policy.py
+++ b/rllib/agents/ddpg/ddpg_tf_policy.py
@@ -175,7 +175,6 @@ def ddpg_actor_critic_loss(policy, model, _, train_batch):
     if twin_q:
         td_error = q_t_selected - q_t_selected_target
         twin_td_error = twin_q_t_selected - q_t_selected_target
-        td_error = td_error + twin_td_error
         if use_huber:
             errors = huber_loss(td_error, huber_threshold) + \
                 huber_loss(twin_td_error, huber_threshold)
diff --git a/rllib/agents/ddpg/ddpg_torch_policy.py b/rllib/agents/ddpg/ddpg_torch_policy.py
--- a/rllib/agents/ddpg/ddpg_torch_policy.py
+++ b/rllib/agents/ddpg/ddpg_torch_policy.py
@@ -124,7 +124,6 @@ def ddpg_actor_critic_loss(policy, model, _, train_batch):
     if twin_q:
         td_error = q_t_selected - q_t_selected_target
         twin_td_error = twin_q_t_selected - q_t_selected_target
-        td_error = td_error + twin_td_error
         if use_huber:
             errors = huber_loss(td_error, huber_threshold) \
                 + huber_loss(twin_td_error, huber_threshold)


Test patch:
diff --git a/rllib/agents/ddpg/tests/test_ddpg.py b/rllib/agents/ddpg/tests/test_ddpg.py
--- a/rllib/agents/ddpg/tests/test_ddpg.py
+++ b/rllib/agents/ddpg/tests/test_ddpg.py
@@ -495,7 +495,6 @@ def _ddpg_loss_helper(self, train_batch, weights, ks, fw, gamma,
 
         td_error = q_t_selected - q_t_selected_target
         twin_td_error = twin_q_t_selected - q_t_selected_target
-        td_error = td_error + twin_td_error
         errors = huber_loss(td_error, huber_threshold) + \
             huber_loss(twin_td_error, huber_threshold)
Repository: ray-project/ray

Issue:
Issue on page /tune/api_docs/execution.html [tune]
Documentation regarding `max_failures`:
```
        max_failures (int): Try to recover a trial at least this many times.
            Ray will recover from the latest checkpoint if present.
            Setting to -1 will lead to infinite recovery retries.
            Setting to 0 will disable retries. Defaults to 3.
```

However, if you look into `tune.run`, the default is `max_failure=0`. I think that the doc is outdated, because the observed behavior is as if `max_failure=0` is true.


Discussion:

Resolving patch:
diff --git a/python/ray/tune/tune.py b/python/ray/tune/tune.py
--- a/python/ray/tune/tune.py
+++ b/python/ray/tune/tune.py
@@ -230,7 +230,7 @@ def run(
         max_failures (int): Try to recover a trial at least this many times.
             Ray will recover from the latest checkpoint if present.
             Setting to -1 will lead to infinite recovery retries.
-            Setting to 0 will disable retries. Defaults to 3.
+            Setting to 0 will disable retries. Defaults to 0.
         fail_fast (bool | str): Whether to fail upon the first error.
             If fail_fast='raise' provided, Tune will automatically
             raise the exception received by the Trainable. fail_fast='raise'


Test patch:
Repository: ray-project/ray

Issue:
[tune] The search algorithms generate more trials than necessary
### What is the problem?

I was under the impression that #10802 should avoid producing too many trials before resources are ready. However, on master, it seems like many trials are generated (especially when using search algorithms) before resources are ready, and this happens far into the experiment.

cc @krfricke 


Discussion:

Resolving patch:
diff --git a/python/ray/tune/trial_runner.py b/python/ray/tune/trial_runner.py
--- a/python/ray/tune/trial_runner.py
+++ b/python/ray/tune/trial_runner.py
@@ -454,7 +454,10 @@ def _get_next_trial(self):
         """
         trials_done = all(trial.is_finished() for trial in self._trials)
         wait_for_trial = trials_done and not self._search_alg.is_finished()
-        self._update_trial_queue(blocking=wait_for_trial)
+        # Only fetch a new trial if we have no pending trial
+        if not any(trial.status == Trial.PENDING for trial in self._trials) \
+           or wait_for_trial:
+            self._update_trial_queue(blocking=wait_for_trial)
         with warn_if_slow("choose_trial_to_run"):
             trial = self._scheduler_alg.choose_trial_to_run(self)
             logger.debug("Running trial {}".format(trial))


Test patch:
diff --git a/python/ray/tune/tests/test_trial_runner.py b/python/ray/tune/tests/test_trial_runner.py
--- a/python/ray/tune/tests/test_trial_runner.py
+++ b/python/ray/tune/tests/test_trial_runner.py
@@ -288,6 +288,44 @@ def on_trial_result(self, trial_runner, trial, result):
         self.assertEqual(trials[0].status, Trial.RUNNING)
         self.assertEqual(runner.trial_executor._committed_resources.cpu, 2)
 
+    def testQueueFilling(self):
+        ray.init(num_cpus=4)
+
+        def f1(config):
+            for i in range(10):
+                yield i
+
+        tune.register_trainable("f1", f1)
+
+        search_alg = BasicVariantGenerator()
+        search_alg.add_configurations({
+            "foo": {
+                "run": "f1",
+                "num_samples": 100,
+                "config": {
+                    "a": tune.sample_from(lambda spec: 5.0 / 7),
+                    "b": tune.sample_from(lambda spec: "long" * 40)
+                },
+                "resources_per_trial": {
+                    "cpu": 2
+                }
+            }
+        })
+
+        runner = TrialRunner(search_alg=search_alg)
+
+        runner.step()
+        runner.step()
+        runner.step()
+        self.assertEqual(len(runner._trials), 3)
+
+        runner.step()
+        self.assertEqual(len(runner._trials), 3)
+
+        self.assertEqual(runner._trials[0].status, Trial.RUNNING)
+        self.assertEqual(runner._trials[1].status, Trial.RUNNING)
+        self.assertEqual(runner._trials[2].status, Trial.PENDING)
+
 
 if __name__ == "__main__":
     import pytest
Repository: jazzband/django-debug-toolbar

Issue:
template context dicts are not showing with django 1.3.0
Here is what I got inspecting all of my pages:

```
{}
{'csrf_token': <django.utils.functional.__proxy__ object at 0x101fba050>}
{}
{'LANGUAGES': '<<languages>>', 'LANGUAGE_BIDI': False, 'LANGUAGE_CODE': 'en-us'}
{'request': '<<request>>'}
{'messages': <django.contrib.messages.storage.cookie.CookieStorage object at 0x101b49650>}
```

It seems the context variables explicitly passed by the view functions are not shown (only the one processed by context processors).



Discussion:
I've seen it also but only when using django.shortcuts.render
If you use django.views.generic.simple.direct_to_template it works.
Seems render does not send the appropriate context from context_processor or so.

There is no difference between django.shortcuts.render() and render_to_response() specifying the context_instance=RequestContext(request).

However, not specifying any context_instance makes DjDT works normally.

Reopened as I closed it by mistake.

Apparently, something alters the contexts in between `TemplateDebugPanel._store_template_info` and `TemplateDebugPanel.content`. I couldn't figure out what it was (need more Django insight), but I could fix it with `kwargs['context'] = kwargs['context'].__copy__()` added as first line to `_store_template_info`. Hope that helps.


Resolving patch:
diff --git a/debug_toolbar/panels/template.py b/debug_toolbar/panels/template.py
--- a/debug_toolbar/panels/template.py
+++ b/debug_toolbar/panels/template.py
@@ -47,6 +47,7 @@ def __init__(self, *args, **kwargs):
         template_rendered.connect(self._store_template_info)
 
     def _store_template_info(self, sender, **kwargs):
+        kwargs['context'] = kwargs['context'].__copy__()
         self.templates.append(kwargs)
 
     def nav_title(self):


Test patch:
Repository: jazzband/django-debug-toolbar

Issue:
Disabling toolbar when settings.TEST seems problematic
I was tripped up by the change made here:

https://github.com/django-debug-toolbar/django-debug-toolbar/pull/167

At issue is that the toolbar doesn't display if settings.TEST is truthy. Is it idiomatic to use settings.TEST to indicate that we're running unit tests? I happen to have used TEST to mean something else, and UNIT_TESTING to mean I'm running unit tests. It would be nice for Django to give us a documented settings flag for doing this (assuming I'm not missing something silly and TEST is that.. in which case this issue is totally misguided).  However, if settings.TEST isn't a setting documented and supported by Django, I find the toolbar using it to suppress the middleware kind of awkward and mysterious. Seems like an exercise best left to the users (who can leave out the middleware during test runs pretty easily).



Discussion:

Resolving patch:
diff --git a/debug_toolbar/middleware.py b/debug_toolbar/middleware.py
--- a/debug_toolbar/middleware.py
+++ b/debug_toolbar/middleware.py
@@ -38,9 +38,6 @@ def show_toolbar(request):
     """
     Default function to determine whether to show the toolbar on a given page.
     """
-    if getattr(settings, 'TEST', False):
-        return False
-
     if request.META.get('REMOTE_ADDR', None) not in settings.INTERNAL_IPS:
         return False
 


Test patch:
diff --git a/tests/tests.py b/tests/tests.py
--- a/tests/tests.py
+++ b/tests/tests.py
@@ -58,10 +58,6 @@ def test_show_toolbar_DEBUG(self):
         with self.settings(DEBUG=False):
             self.assertFalse(show_toolbar(self.request))
 
-    def test_show_toolbar_TEST(self):
-        with self.settings(TEST=True):
-            self.assertFalse(show_toolbar(self.request))
-
     def test_show_toolbar_INTERNAL_IPS(self):
         with self.settings(INTERNAL_IPS=[]):
             self.assertFalse(show_toolbar(self.request))
Repository: jazzband/django-debug-toolbar

Issue:
Consider removing support for logbook
It would fit better in a third-party panel.



Discussion:

Resolving patch:
diff --git a/debug_toolbar/panels/logging.py b/debug_toolbar/panels/logging.py
--- a/debug_toolbar/panels/logging.py
+++ b/debug_toolbar/panels/logging.py
@@ -68,40 +68,13 @@ def emit(self, record):
         self.collector.add_record(record)
 
 
+# We don't use enable/disable_instrumentation because logging is global.
+# We can't add thread-local logging handlers. Hopefully logging is cheap.
+
 collector = LogCollector()
 logging_handler = ThreadTrackingHandler(collector)
 logging.root.setLevel(logging.NOTSET)
-logging.root.addHandler(logging_handler)  # register with logging
-
-# We don't use enable/disable_instrumentation because we can't make these
-# functions thread-safe and (hopefully) logging isn't too expensive.
-
-try:
-    import logbook
-    logbook_supported = True
-except ImportError:
-    # logbook support is optional, so fail silently
-    logbook_supported = False
-
-if logbook_supported:
-    class LogbookThreadTrackingHandler(logbook.handlers.Handler):
-        def __init__(self, collector):
-            logbook.handlers.Handler.__init__(self, bubble=True)
-            self.collector = collector
-
-        def emit(self, record):
-            record = {
-                'message': record.message,
-                'time': record.time,
-                'level': record.level_name,
-                'file': record.filename,
-                'line': record.lineno,
-                'channel': record.channel,
-            }
-            self.collector.add_record(record)
-
-    logbook_handler = LogbookThreadTrackingHandler(collector)
-    logbook_handler.push_application()  # register with logbook
+logging.root.addHandler(logging_handler)
 
 
 class LoggingPanel(Panel):


Test patch:
Repository: jazzband/django-debug-toolbar

Issue:
Allow profiler max depth to be set in settings
Sometimes it's helpful to have a max depth of more than 10, which is currently hard-coded in the profiling panel.



Discussion:
Yes. The patch needs documentation.


Resolving patch:
diff --git a/debug_toolbar/panels/profiling.py b/debug_toolbar/panels/profiling.py
--- a/debug_toolbar/panels/profiling.py
+++ b/debug_toolbar/panels/profiling.py
@@ -3,6 +3,7 @@
 from django.utils.translation import ugettext_lazy as _
 from django.utils.safestring import mark_safe
 from debug_toolbar.panels import Panel
+from debug_toolbar import settings as dt_settings
 
 import cProfile
 from pstats import Stats
@@ -152,6 +153,9 @@ def process_response(self, request, response):
         root = FunctionCall(self.stats, self.stats.get_root_func(), depth=0)
 
         func_list = []
-        self.add_node(func_list, root, 10, root.stats[3] / 8)
+        self.add_node(func_list,
+                      root,
+                      dt_settings.CONFIG['PROFILER_MAX_DEPTH'],
+                      root.stats[3] / 8)
 
         self.record_stats({'func_list': func_list})
diff --git a/debug_toolbar/settings.py b/debug_toolbar/settings.py
--- a/debug_toolbar/settings.py
+++ b/debug_toolbar/settings.py
@@ -33,6 +33,7 @@
         'debug_toolbar',
         'django',
     ),
+    'PROFILER_MAX_DEPTH': 10,
     'SHOW_TEMPLATE_CONTEXT': True,
     'SQL_WARNING_THRESHOLD': 500,   # milliseconds
 }


Test patch:
Repository: jazzband/django-debug-toolbar

Issue:
Removed In Django 1.9 Warning: url tag from future
The full warning:

/Users/JJZ/.virtualenvs/athletesunited/lib/python3.4/site-packages/django/templatetags/future.py:25: RemovedInDjango19Warning: Loading the `url` tag from the `future` library is deprecated and will be removed in Django 1.9. Use the default `url` tag instead.
  RemovedInDjango19Warning)

I've checked through all my code and I can't find a trace of where I do: {% load url from future %}

Now I've moved onto trying to help third party developers fix their code because I'm getting pretty tired of the warnings popping up. When I ran a search on your code base:

https://github.com/django-debug-toolbar/django-debug-toolbar/search?utf8=%E2%9C%93&q=%7B%25+load+url+from+future+%25%7D

I see that you have used this, this is from way back in Django 1.3 and it is being removed in 1.9 so if you all would, could you please go through clear this all up and let me know when the commit has been pushed so I can pull the updates?

Thanks a lot,

JJ

PS: https://docs.djangoproject.com/en/1.8/releases/1.7/#loading-ssi-and-url-template-tags-from-future-library



Discussion:

Resolving patch:
diff --git a/debug_toolbar/compat.py b/debug_toolbar/compat.py
--- a/debug_toolbar/compat.py
+++ b/debug_toolbar/compat.py
@@ -5,6 +5,7 @@
 debug_toolbar.
 """
 
+import django
 from django.conf import settings
 from django.core.exceptions import ImproperlyConfigured
 
@@ -92,3 +93,13 @@ def get_template_context_processors():
     else:  # Django < 1.8
         context_processors = get_standard_processors()
     return context_processors
+
+if django.VERSION[:2] < (1, 5):
+    # If the user is using Django < 1.5, then load up the url tag
+    # from future. Otherwise use the normal one. The purpose of this
+    # is to get the url template tag that supports context variables
+    # for the first argument, yet won't raise a deprecation warning
+    # about importing it from future.
+    from django.templatetags.future import url
+else:
+    from django.template.defaulttags import url  # NOQA
diff --git a/debug_toolbar/templatetags/__init__.py b/debug_toolbar/templatetags/__init__.py
new file mode 100644
diff --git a/debug_toolbar/templatetags/compat.py b/debug_toolbar/templatetags/compat.py
new file mode 100644
--- /dev/null
+++ b/debug_toolbar/templatetags/compat.py
@@ -0,0 +1,10 @@
+from django.template.base import Library
+
+from ..compat import url as url_compat
+
+register = Library()
+
+
+@register.tag
+def url(parser, token):
+    return url_compat(parser, token)


Test patch:
Repository: jazzband/django-debug-toolbar

Issue:
cache.clear() not available inside a view function
cache.clear() raises NotImplementedError during request/view processing between debug toolbar middleware request processing and debug toolbar middleware response processing.

This is a copy of my message in closed issue #570:

It took me all of 10 minutes to find out why this was happening in my project, inside a view function:
- the cache panel replaces the default cache (and the get_cache) function during the request middleware processing with CacheStatTracker/a function returning CacheStatTracker
- processing continues until the actual view function is called
- cache.clear() is called inside the view function, on CacheStatTracker
- CacheStatTracker has no clear() method, but has inherited the clear() method from BaseCache
- clear() is called on BaseCache
- BaseCache raises NotImplementedError

You won't find this occurring outside a view function, because on the way back out, the processing passes through the debug toolbar middleware again, this time calling disable_instrumentation on the cache panel, which replaces the cache/get_cache with the original values again.



Discussion:
If I understand correctly, the fix is simply to proxy `clear` in `CacheStatTracker`?

Yes that should fix it.

Excellent.

PS: sorry for taking three months to answer — I don't look at this repository very often.


Resolving patch:
diff --git a/debug_toolbar/panels/cache.py b/debug_toolbar/panels/cache.py
--- a/debug_toolbar/panels/cache.py
+++ b/debug_toolbar/panels/cache.py
@@ -81,6 +81,10 @@ def set(self, *args, **kwargs):
     def delete(self, *args, **kwargs):
         return self.cache.delete(*args, **kwargs)
 
+    @send_signal
+    def clear(self, *args, **kwargs):
+        return self.cache.clear(*args, **kwargs)
+
     @send_signal
     def has_key(self, *args, **kwargs):
         # Ignore flake8 rules for has_key since we need to support caches
@@ -148,6 +152,7 @@ def __init__(self, *args, **kwargs):
             ('get', 0),
             ('set', 0),
             ('delete', 0),
+            ('clear', 0),
             ('get_many', 0),
             ('set_many', 0),
             ('delete_many', 0),


Test patch:
Repository: jazzband/django-debug-toolbar

Issue:
The "compat" templatetag library collides with other apps
[django-imagekit](https://github.com/matthewwithanm/django-imagekit) has shipped a `compat` file in it's own templatetags [since 2012](https://github.com/matthewwithanm/django-imagekit/commits/develop/imagekit/templatetags/compat.py). It isn't actually exposed as a template library, though.

The introduction of `debug_toolbar.templatetags.compat` in 5a9bd46f8624320346d829687d3709b0a9beb580 means that there is now an order of precedence in INSTALLED_APPS issue; given:

```
INSTALLED_APPS = (
    'debug_toolbar',
    'imagekit',
)
```

everything is fine.

Swap those values around so that `imagekit` comes first, and debug_toolbar causes:

```
'compat' is not a valid tag library: Template library imagekit.templatetags.compat does not have a variable named 'register'
```

It would seem prudent to me to have the `compat` module namespaced as `debugtoolbar_compat` or such in templates.



Discussion:
Yes, this is a sensible improvement.

+1

If someone feels like putting together a pull request, the patch shouldn't be too hard.

I'll get this done tomorrow.


Resolving patch:
diff --git a/debug_toolbar/templatetags/compat.py b/debug_toolbar/templatetags/debug_toolbar_compat.py
similarity index 100%
rename from debug_toolbar/templatetags/compat.py
rename to debug_toolbar/templatetags/debug_toolbar_compat.py


Test patch: