internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: django-haystack/django-haystack

Issue:
build_params in solr_backend.py ignores **kwargs leading to unnecessarily large solr responses
I noticed that when I do for example:

```
SearchQuerySet().filter(…).values_list('pk', flat=True)
```

Django would request the FULL documents from the SOLR server, although SOLR supports specifying just the fields you want with the `fl` query parameter. This is quite wasteful if all you want is a list of pks.

I traced it to https://github.com/toastdriven/django-haystack/blob/master/haystack/backends/solr_backend.py#L628 where `build_params` ignores `kwargs` completely. In this case, `kwargs` contains `set(['django_ct', 'pk', 'django_id', 'score', 'id'])` which is EXACTLY what should be passed to `SolrSearchBackend.search()` to get the desired behaviour.

In short, having `build_params()` merge `kwargs` with `search_kwargs` and returning that would solve this nicely.

Is the reason for this the caching, or was it an oversight? It would be great to request only the data that we actually need from the solr server, and not the complete documents every single time.



Discussion:
I'm pretty sure this was a regression – I thought I confirmed the fl= usage
when writing the values / values_list support originally.

I'm +1 on making this change and adding a test to confirm that it's present
in the generated query.

Chris


Resolving patch:
diff --git a/haystack/backends/elasticsearch_backend.py b/haystack/backends/elasticsearch_backend.py
--- a/haystack/backends/elasticsearch_backend.py
+++ b/haystack/backends/elasticsearch_backend.py
@@ -911,6 +911,10 @@ def run(self, spelling_query=None, **kwargs):
         """Builds and executes the query. Returns a list of search results."""
         final_query = self.build_query()
         search_kwargs = self.build_params(spelling_query, **kwargs)
+
+        if kwargs:
+            search_kwargs.update(kwargs)
+
         results = self.backend.search(final_query, **search_kwargs)
         self._results = results.get('results', [])
         self._hit_count = results.get('hits', 0)
diff --git a/haystack/backends/solr_backend.py b/haystack/backends/solr_backend.py
--- a/haystack/backends/solr_backend.py
+++ b/haystack/backends/solr_backend.py
@@ -690,6 +690,9 @@ def run(self, spelling_query=None, **kwargs):
         final_query = self.build_query()
         search_kwargs = self.build_params(spelling_query, **kwargs)
 
+        if kwargs:
+            search_kwargs.update(kwargs)
+
         results = self.backend.search(final_query, **search_kwargs)
         self._results = results.get('results', [])
         self._hit_count = results.get('hits', 0)


Test patch:
Repository: django-haystack/django-haystack

Issue:
Fix imports for elasticsearch 1.8+
`elasticsearch.helpers.bulk_index` was removed In this commit:

https://github.com/elastic/elasticsearch-py/commit/5d4640d4c2cfd52aa92121fe41bbb383945cf6d3



Discussion:

Resolving patch:
diff --git a/haystack/backends/elasticsearch_backend.py b/haystack/backends/elasticsearch_backend.py
--- a/haystack/backends/elasticsearch_backend.py
+++ b/haystack/backends/elasticsearch_backend.py
@@ -22,7 +22,12 @@
 
 try:
     import elasticsearch
-    from elasticsearch.helpers import bulk_index
+    try:
+        # let's try this, for elasticsearch > 1.7.0
+        from elasticsearch.helpers import bulk
+    except ImportError:
+        # let's try this, for elasticsearch <= 1.7.0
+        from elasticsearch.helpers import bulk_index as bulk
     from elasticsearch.exceptions import NotFoundError
 except ImportError:
     raise MissingDependency("The 'elasticsearch' backend requires the installation of 'elasticsearch'. Please refer to the documentation.")
@@ -188,7 +193,7 @@ def update(self, index, iterable, commit=True):
                     }
                 })
 
-        bulk_index(self.conn, prepped_docs, index=self.index_name, doc_type='modelresult')
+        bulk(self.conn, prepped_docs, index=self.index_name, doc_type='modelresult')
 
         if commit:
             self.conn.indices.refresh(index=self.index_name)


Test patch:
Repository: django-haystack/django-haystack

Issue:
Limit django versions in setup.py
Since the currently released version of haystack is incompatible with django 1.9 people have to manually install django 1.8 it would be nice if pip would handle this automatically.

The same problem will probably also arise when django 1.10 is released.
For the next release please limit django versions to >= 1.8 and < 1.10

I'm not sure when you're planning a new release that supports 1.9. But if it's still a while, can you please release a new version where you just limit the django version?



Discussion:
Any updates on this?
Really want to use Django 1.9 =)

Please test the master branch - the release has been waiting on people with time to test & review it.

Hey Chris, sorry, but I think we are going nowhere asking people to test master. Would it be possible for you to release at least a rc version?

For what it's worth, I've tested the master branch with my [drf-haystack](https://github.com/inonit/drf-haystack) library and it seems to be working good (at least it's passing my test suite). I'm waiting to release a new version until there's a new django-haystack version, so at least an rc version would be fantastic!

Since Django 1.10 is nigh, I've added a Travis target which tests the alpha builds but marked it as allow failure. https://travis-ci.org/acdha/django-haystack/jobs/133485740#L1676-L1682 suggests we'll need to deal with the deferred logic changing.

@acdha You still didn't fix the original issue. Whenever you install django-haystack it automatically uses the latest version of django. Since it's incompatible with 1.10 all setups from that point in time will fail. You need to limit the django version in setup.py.

A lower and upper limit should be added, to ensure people actually have compatible django versions


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -12,7 +12,8 @@
     from setuptools import setup
 
 install_requires = [
-    'Django',
+    'Django>=1.8',
+    'Django<1.10',
 ]
 
 tests_require = [


Test patch:
Repository: django-haystack/django-haystack

Issue:
Problem with pagination when missing ``q`` GET parameter for ``SearchView``
* [x] Tested with the latest Haystack release
* [x] Tested with the current Haystack master branch

## Actual behaviour

Suppose ``haystack.generic_views.SearchView`` is bound to ``/search`` url. 

When I enter the search url ``/search`` I get a paginated list of all indexed entities. When I enter ``/search?page=2`` I get error stating that no such page exists when in fact there is more than a single page available. 

After some digging I found that different code paths are executed in both cases. 

When I enter ``/search`` following things happen: 
*  ``SearchForm.is_valid`` returns ``False`` (form instance wasn't passed ``data`` parameter, and hence ``is_bound`` returns true, which makes the form invalid)
* ``SearchView.form_invalid`` is called and it returns full queryset
* Full queryset is paginated 

While when I enter ``/search?page=2``: 
* ``SearchForm.is_valid`` returns ``True`` 
* ``SearchView.form_valid`` is called
* which returns ``form.search()`` which in case of missing ``q`` parameter returns ``self.no_query_found()`` (which is empty by default)

## Expected behaviour 

I'd like the behavior to be consistent, either return (whole) queryset or return empty queryset in both cases. 

Probably it would be better to always trigger second behavior, as then it can be customized by overriding ``self.no_query_found()``. If that is the case, following [patch should fix this case](https://github.com/django-haystack/django-haystack/compare/master...jbzdak:jbzdak-patch?expand=1)

## Configuration

* Operating system version: Debian Linux (on docker that still uses Debian Linux Image)
* Search engine version: Most recent stable Whoosh 
* Python version: 3.5
* Django version: 1.10.4
* Haystack version: Both most recent stable and master 


Discussion:

Resolving patch:
diff --git a/haystack/generic_views.py b/haystack/generic_views.py
--- a/haystack/generic_views.py
+++ b/haystack/generic_views.py
@@ -58,7 +58,7 @@ def get_form_kwargs(self):
         Returns the keyword arguments for instantiating the form.
         """
         kwargs = {'initial': self.get_initial()}
-        if self.request.method == 'GET' and self.request.GET:
+        if self.request.method == 'GET':
             kwargs.update({
                 'data': self.request.GET,
             })


Test patch:
Repository: django-haystack/django-haystack

Issue:
SearchQuerySet is instantiated at startup
* [X] Tested with the latest Haystack release
* [ ] Tested with the current Haystack master branch

## Expected behaviour
No backend should be run during Django check or startup.

## Actual behaviour
When using a SearchView, the SearchQuerySet is instantiated right away and messes with the correct language.

## Steps to reproduce the behaviour

1. Drop a `print`/`breakpoint`/`assert` in `ElasticsearchSearchBackend.__init__()`
1. Run `manage.py runserver`
1. Extra: if using AldrynSearch with a multilanguage setup, the wrong index is taken.


## Configuration

* Operating system version: Fedora 28
* Search engine version: Elasticsearch Version: `2.4.6`, _Build: 5376dca/2017-07-18T12:17:44Z, JVM: 1.8.0_172_
* Python version: `3.5.5` _[GCC 8.0.1 20180324 (Red Hat 8.0.1-0.20)] on linux_
* Django version: `1.11.13`
* Haystack version: `2.8.1`


Discussion:

Resolving patch:
diff --git a/haystack/generic_views.py b/haystack/generic_views.py
--- a/haystack/generic_views.py
+++ b/haystack/generic_views.py
@@ -44,7 +44,6 @@ class SearchMixin(MultipleObjectMixin, FormMixin):
     template_name = "search/search.html"
     load_all = True
     form_class = ModelSearchForm
-    queryset = SearchQuerySet()
     context_object_name = None
     paginate_by = RESULTS_PER_PAGE
     paginate_orphans = 0
@@ -54,6 +53,11 @@ class SearchMixin(MultipleObjectMixin, FormMixin):
     search_field = "q"
     object_list = None
 
+    def get_queryset(self):
+        if self.queryset is None:
+            self.queryset = SearchQuerySet()
+        return self.queryset
+
     def get_form_kwargs(self):
         """
         Returns the keyword arguments for instantiating the form.


Test patch:
Repository: django-haystack/django-haystack

Issue:
Compatibility with django 3
Using django-haystack with django 3 leads to a bunch of warnings similar to

```
  /usr/local/lib/python3.8/site-packages/haystack/forms.py:25: RemovedInDjango40Warning: django.utils.translation.ugettext_lazy() is deprecated in favor of django.utils.translation.gettext_lazy().
    label=_("Search"),
```

which is caused by

https://github.com/django-haystack/django-haystack/blob/1ef7e65e69025bdef0756f1f2e3679499c4e8e25/haystack/forms.py#L5




Discussion:

Resolving patch:
diff --git a/haystack/forms.py b/haystack/forms.py
--- a/haystack/forms.py
+++ b/haystack/forms.py
@@ -2,7 +2,7 @@
 from django import forms
 from django.utils.encoding import smart_text
 from django.utils.text import capfirst
-from django.utils.translation import ugettext_lazy as _
+from django.utils.translation import gettext_lazy as _
 
 from haystack import connections
 from haystack.constants import DEFAULT_ALIAS
diff --git a/haystack/panels.py b/haystack/panels.py
--- a/haystack/panels.py
+++ b/haystack/panels.py
@@ -3,7 +3,7 @@
 
 from debug_toolbar.panels import DebugPanel
 from django.template.loader import render_to_string
-from django.utils.translation import ugettext_lazy as _
+from django.utils.translation import gettext_lazy as _
 
 from haystack import connections
 


Test patch:
Repository: django-haystack/django-haystack

Issue:
Updating whoosh index with --workers > 1 misses data
* [ ] Tested with the latest Haystack release
* [x] Tested with the current Haystack master branch

## Expected behaviour
Index item count should match the number of indexed objects

## Actual behaviour
Index item count is less than the number of indexed objects

## Description of actual problem

Using Whoosh as a backend, update_index:

https://github.com/django-haystack/django-haystack/blob/8ab14a273d8f438fe3542d988954e41bddc13fc8/haystack/management/commands/update_index.py#L119

calls into the Haystack Whoosh backend here:

https://github.com/django-haystack/django-haystack/blob/8ab14a273d8f438fe3542d988954e41bddc13fc8/haystack/backends/whoosh_backend.py#L247

which uses an AsyncWriter,

https://github.com/django-haystack/django-haystack/blob/8ab14a273d8f438fe3542d988954e41bddc13fc8/haystack/backends/whoosh_backend.py#L252

and later commits.

https://github.com/django-haystack/django-haystack/blob/8ab14a273d8f438fe3542d988954e41bddc13fc8/haystack/backends/whoosh_backend.py#L284-L286

For reference, here's whoosh.writing.AsyncWriter.commit:

https://github.com/mchaput/whoosh/blob/5421f1ab3bb802114105b3181b7ce4f44ad7d0bb/src/whoosh/writing.py#L1044-L1049

If the AsyncWriter managed to grab the lock on creation, commit commits and is done.

If the AsyncWriter did not manage to grab the lock on creation, a thread is spun up on `commit()` to try to grab the lock and flush out the writes. Nothing `join()`s that thread, so if the worker exits before its thread can make the writes, they are lost. There is likely also a race between the worker exiting and the thread writing.

I'll send a PR later, but it's going to be basically this:

https://github.com/django-haystack/django-haystack/blob/8ab14a273d8f438fe3542d988954e41bddc13fc8/haystack/backends/whoosh_backend.py#L284-L286

follow the `commit()` call with:
```python
if writer.ident is not None:
    writer.join()
```

Another plan might be to queue up all the threads and join at the end of the worker's tasks.


Discussion:

Resolving patch:
diff --git a/haystack/backends/whoosh_backend.py b/haystack/backends/whoosh_backend.py
--- a/haystack/backends/whoosh_backend.py
+++ b/haystack/backends/whoosh_backend.py
@@ -284,6 +284,8 @@ def update(self, index, iterable, commit=True):
         if len(iterable) > 0:
             # For now, commit no matter what, as we run into locking issues otherwise.
             writer.commit()
+            if writer.ident is not None:
+                writer.join()
 
     def remove(self, obj_or_string, commit=True):
         if not self.setup_complete:


Test patch:
Repository: django-haystack/django-haystack

Issue:
Update extras_require in setup.py to support Elasticsearch 7 
Pipenv failed to generate lock file with `elasticsearch>=7.0.0` due to the bounded version in `setup.py`. Should adjust bounded version in `extras_require` so that `Pipfile.lock` can be generated


![](https://user-images.githubusercontent.com/16464044/149522995-848940d1-bc6e-4e90-b06d-947b6cdd115b.png)



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -55,7 +55,7 @@
     install_requires=install_requires,
     tests_require=tests_require,
     extras_require={
-        "elasticsearch": ["elasticsearch>=5,<6"],
+        "elasticsearch": ["elasticsearch>=5,<8"],
     },
     test_suite="test_haystack.run_tests.run_all",
 )


Test patch:
Repository: secdev/scapy

Issue:
Tcp FlagsField length not respecting RFC
Could not use "N" (Nonce in whireshark") flag in TCP packet. Bit added to the `FlagsField` in `layers/inet.py`.

Before:

``` python
class TCP(Packet):
    name = "TCP"
    fields_desc = [ ShortEnumField("sport", 20, TCP_SERVICES),
                    ShortEnumField("dport", 80, TCP_SERVICES),
                    IntField("seq", 0),
                    IntField("ack", 0),
                    BitField("dataofs", None, 4),
                    BitField("reserved", 0, 4),
                    FlagsField("flags", 0x2, 8, "FSRPAUEC"),
                    ShortField("window", 8192),
                    XShortField("chksum", None),
                    ShortField("urgptr", 0),
                    TCPOptionsField("options", {}) ]
```

After:

``` python
class TCP(Packet):
    name = "TCP"
    fields_desc = [ ShortEnumField("sport", 20, TCP_SERVICES),
                    ShortEnumField("dport", 80, TCP_SERVICES),
                    IntField("seq", 0),
                    IntField("ack", 0),
                    BitField("dataofs", None, 4),
                    BitField("reserved", 0, 3),
                    FlagsField("flags", 0x2, 9, "FSRPAUECN"),
                    ShortField("window", 8192),
                    XShortField("chksum", None),
                    ShortField("urgptr", 0),
                    TCPOptionsField("options", {}) ]
```



Discussion:

Resolving patch:
diff --git a/scapy/layers/inet.py b/scapy/layers/inet.py
--- a/scapy/layers/inet.py
+++ b/scapy/layers/inet.py
@@ -440,8 +440,8 @@ class TCP(Packet):
                     IntField("seq", 0),
                     IntField("ack", 0),
                     BitField("dataofs", None, 4),
-                    BitField("reserved", 0, 4),
-                    FlagsField("flags", 0x2, 8, "FSRPAUEC"),
+                    BitField("reserved", 0, 3),
+                    FlagsField("flags", 0x2, 9, "FSRPAUECN"),
                     ShortField("window", 8192),
                     XShortField("chksum", None),
                     ShortField("urgptr", 0),


Test patch:
Repository: secdev/scapy

Issue:
Packet.command() behavior with Net6
The command() method is cool, yet it does not work as expected with IPv6: the resulting string is not valid.

``` python
>>> p = IPv6(dst="www.kame.net")/ICMPv6EchoRequest()
>>> p.command()
'IPv6(dst=<Net6 www.kame.net>)/ICMPv6EchoRequest()'
```

``` python
>>> p = IP(dst="www.kame.net")/ICMP()
>>> p.command()
"IP(dst=Net('www.kame.net'))/ICMP()"
```

Packet.command() behavior with Net6
The command() method is cool, yet it does not work as expected with IPv6: the resulting string is not valid.

``` python
>>> p = IPv6(dst="www.kame.net")/ICMPv6EchoRequest()
>>> p.command()
'IPv6(dst=<Net6 www.kame.net>)/ICMPv6EchoRequest()'
```

``` python
>>> p = IP(dst="www.kame.net")/ICMP()
>>> p.command()
"IP(dst=Net('www.kame.net'))/ICMP()"
```



Discussion:

Resolving patch:
diff --git a/scapy/layers/inet6.py b/scapy/layers/inet6.py
--- a/scapy/layers/inet6.py
+++ b/scapy/layers/inet6.py
@@ -177,7 +177,7 @@ def rec(n, l):
         return iter(rec(0, ['']))
 
     def __repr__(self):
-        return "<Net6 %s>" % self.repr
+        return "Net6(%r)" % self.repr
 
 
 


Test patch:
Repository: secdev/scapy

Issue:
IKEv2 support is broken
This issue is related to PR #52.

``` python
>>> load_contrib("ikev2"); p = IP()/UDP()/IKEv2(init_SPI=RandString(8), exch_type=34)/IKEv2_payload_SA(prop=IKEv2_payload_Proposal())
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "scapy/base_classes.py", line 223, in __call__
    i.__init__(*args, **kargs)
  File "scapy/packet.py", line 78, in __init__
    self.overload_fields = self._overload_fields
  File "scapy/packet.py", line 211, in __setattr__
    return object.__setattr__(self, attr, val)
AttributeError: 'IKEv2' object attribute 'overload_fields' is read-only
```

This should be added to UT.



Discussion:

Resolving patch:
diff --git a/scapy/contrib/ikev2.py b/scapy/contrib/ikev2.py
--- a/scapy/contrib/ikev2.py
+++ b/scapy/contrib/ikev2.py
@@ -343,7 +343,7 @@ class IKEv2_payload_Encrypted(IKEv2_class):
         IKEv2_payload_type_overload[globals()[name]] = {"next_payload": i}
 
 del i, payloadname, name
-IKEv2_class.overload_fields = IKEv2_payload_type_overload.copy()
+IKEv2_class._overload_fields = IKEv2_payload_type_overload.copy()
 
 split_layers(UDP, ISAKMP, sport=500)
 split_layers(UDP, ISAKMP, dport=500)


Test patch:
Repository: secdev/scapy

Issue:
Export Historyfile
 I want to put all the console history files in one place

 In mysql you can do
 export MSQL_HISTFILE=~/.history/mysql
 and the history file is created and stored

 I can so this with bash,python,postgres

 How can I do it with scapy in linux using the $HOME/.bashrc

 I've tried

 export PYTHONSCAPY_HISTORY=~/.history/scapy
 export PYTHONSCAPY_HIST=~/.history/scapy
 export SCAPY_HIST=~/.history/scapy
 export SCAPY_HISTORY=~/.history/scapy

None of these work Im told by phil@secdev.org its set by

'>>>' conf.histfile = 

is there any way to set it as a environment varible or someone make a patch



Discussion:

Resolving patch:
diff --git a/scapy/config.py b/scapy/config.py
--- a/scapy/config.py
+++ b/scapy/config.py
@@ -346,7 +346,9 @@ class Conf(ConfClass):
     L2socket = None
     L2listen = None
     min_pkt_size = 60
-    histfile = os.path.join(os.path.expanduser("~"), ".scapy_history")
+    histfile = os.getenv('SCAPY_HISTFILE',
+                         os.path.join(os.path.expanduser("~"),
+                                      ".scapy_history"))
     padding = 1
     except_filter = ""
     debug_match = 0


Test patch:
Repository: secdev/scapy

Issue:
Unable to Accurately Set Time Fields in NTP Layer
I'm trying to modify the time fields in the NTP layer, and not having any luck. I've tried setting packet.recv to an integer value and a string representation of time, but no matter what I try, when I send the packet, this field is always set to the default 0. Is there any example code that I just missed that would show me how to do this? Thank you in advance.



Discussion:

Resolving patch:
diff --git a/scapy/layers/ntp.py b/scapy/layers/ntp.py
--- a/scapy/layers/ntp.py
+++ b/scapy/layers/ntp.py
@@ -8,6 +8,7 @@
 """
 
 import time
+import datetime
 from scapy.packet import *
 from scapy.fields import *
 from scapy.layers.inet import UDP
@@ -29,9 +30,11 @@ def i2repr(self, pkt, val):
         return time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime(val-_NTP_BASETIME))
 
     def any2i(self, pkt, val):
-        if type(val) is str:
-            return int(time.mktime(time.strptime(val))) + _NTP_BASETIME + 3600 # XXX
-        return FixedPointField.any2i(self,pkt,val)
+        if isinstance(val, basestring):
+            val = int(time.mktime(time.strptime(val))) + _NTP_BASETIME
+        elif isinstance(val, datetime.datetime):
+            val = int(val.strftime("%s")) + _NTP_BASETIME
+        return FixedPointField.any2i(self, pkt, val)
     
     def i2m(self, pkt, val):
         if val is None:


Test patch:
Repository: secdev/scapy

Issue:
Scapy (re.graph) Issue
Hi,

Here is the Scapy Code (Version 2.2.0):

```
res,unans = traceroute([“12.0.2.2”],dport=[80,443],maxttl=20,retry=-2)
res.graph(target="> /tmp/graph.svg")
```

Error:
![index](https://cloud.githubusercontent.com/assets/2124883/15610395/505ec99c-2425-11e6-9a0f-21cb2a1c2433.png)

Is it not possible to only use the IP address for graphing purposes. I have internal network diagram and want to use Scapy in this. No DNS implementation in the current network. 

Please advice.

Kind regards,
Aubrey



Discussion:

Resolving patch:
diff --git a/scapy/layers/inet.py b/scapy/layers/inet.py
--- a/scapy/layers/inet.py
+++ b/scapy/layers/inet.py
@@ -995,7 +995,8 @@ def _wrap_data(ts_tuple, wrap_seconds=2000):
 
 ### Create a new packet list
 class TracerouteResult(SndRcvList):
-    __slots__ = ["graphdef", "graphASres", "padding", "hloc", "nloc"]
+    __slots__ = ["graphdef", "graphpadding", "graphASres", "padding", "hloc",
+                 "nloc"]
     def __init__(self, res=None, name="Traceroute", stats=None):
         PacketList.__init__(self, res, name, stats)
         self.graphdef = None
@@ -1273,7 +1274,8 @@ def make_graph(self,ASres=None,padding=0):
         bhip = {}
         for rtk in rt:
             trace = rt[rtk]
-            for n in xrange(min(trace), max(trace)):
+            max_trace = max(trace)
+            for n in xrange(min(trace), max_trace):
                 if not trace.has_key(n):
                     trace[n] = unknown_label.next()
             if not ports_done.has_key(rtk):
@@ -1288,7 +1290,7 @@ def make_graph(self,ASres=None,padding=0):
                 ips[bh] = None
                 bhip[rtk[1]] = bh
                 bh = '"%s"' % bh
-                trace[max(k)+1] = bh
+                trace[max_trace + 1] = bh
                 blackholes.append(bh)
     
         # Find AS numbers


Test patch: