internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
Database [performance] for group_extras and resource
Related to #3511

### CKAN Version if known (or site URL)
2.8.3

### Please describe the expected behaviour
Fetching from `group_extra` by `group_id` or from `resource` by `package_id` is fast.

### Please describe the actual behaviour
With enough data, both become slow. The former is especially visible when fetching `organization_list`.

### What steps can be taken to reproduce the issue? 
Insert a lot of organizations with a translated `ckanext-fluent` `title_translated` field, then attempt to get a list of organizations with translated names.

### Proposed fix
Add database indices for `group_extra (group_id)` and `resource (package_id)`. These speed up the operations considerably.


Discussion:
I would've included a PR, but master has transitioned to alembic and I haven't had the time yet to look into it.
The additions that you propose sound great to have on master (2.9) so a PR is most welcome. On previous releases we can suggest the change in the CHANGELOG so people can run it manually if needed.

Resolving patch:
diff --git a/ckan/migration/versions/091_0ffc0b277141_group_extra_group_id_index.py b/ckan/migration/versions/091_0ffc0b277141_group_extra_group_id_index.py
new file mode 100644
--- /dev/null
+++ b/ckan/migration/versions/091_0ffc0b277141_group_extra_group_id_index.py
@@ -0,0 +1,25 @@
+# encoding: utf-8
+
+"""group_extra group_id index
+
+Revision ID: 0ffc0b277141
+Revises: 980dcd44de4b
+Create Date: 2019-09-11 12:16:42.792661
+
+"""
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision = u'0ffc0b277141'
+down_revision = u'980dcd44de4b'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.create_index(u'idx_group_extra_group_id', u'group_extra', [u'group_id'])
+
+
+def downgrade():
+    op.drop_index(u'idx_group_extra_group_id')
diff --git a/ckan/migration/versions/092_01afcadbd8c0_resource_package_id_index.py b/ckan/migration/versions/092_01afcadbd8c0_resource_package_id_index.py
new file mode 100644
--- /dev/null
+++ b/ckan/migration/versions/092_01afcadbd8c0_resource_package_id_index.py
@@ -0,0 +1,26 @@
+# encoding: utf-8
+
+"""resource package_id index
+
+Revision ID: 01afcadbd8c0
+Revises: 0ffc0b277141
+Create Date: 2019-09-11 12:16:53.937813
+
+"""
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision = u'01afcadbd8c0'
+down_revision = u'0ffc0b277141'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.create_index(u'idx_package_resource_package_id', u'resource',
+                    [u'package_id'])
+
+
+def downgrade():
+    op.drop_index(u'idx_package_resource_package_id')


Test patch:
Repository: ckan/ckan

Issue:
RFE: allow Reply-To header on emails
### CKAN Version if known (or site URL)

2.7.3

### Please describe the expected behaviour

We would like to send system emails that come from a real address (so we can catch bounces etc), but which don't reply to a real address when used by humans (ie use a 'no-reply' address as the Reply-To header).

### Please describe the actual behaviour

Only the 'From' address is configurable.



Discussion:

Resolving patch:
diff --git a/ckan/lib/mailer.py b/ckan/lib/mailer.py
--- a/ckan/lib/mailer.py
+++ b/ckan/lib/mailer.py
@@ -37,6 +37,7 @@ def _mail_recipient(recipient_name, recipient_email,
         headers = {}
 
     mail_from = config.get('smtp.mail_from')
+    reply_to = config.get('smtp.reply_to')
     msg = MIMEText(body.encode('utf-8'), 'plain', 'utf-8')
     for k, v in headers.items():
         if k in msg.keys():
@@ -50,6 +51,8 @@ def _mail_recipient(recipient_name, recipient_email,
     msg['To'] = Header(recipient, 'utf-8')
     msg['Date'] = Utils.formatdate(time())
     msg['X-Mailer'] = "CKAN %s" % ckan.__version__
+    if reply_to and reply_to != '':
+        msg['Reply-to'] = reply_to
 
     # Send the email using Python's smtplib.
     smtp_connection = smtplib.SMTP()


Test patch:
diff --git a/ckan/tests/lib/test_mailer.py b/ckan/tests/lib/test_mailer.py
--- a/ckan/tests/lib/test_mailer.py
+++ b/ckan/tests/lib/test_mailer.py
@@ -233,3 +233,26 @@ def test_bad_smtp_host(self):
                       'headers': {'header1': 'value1'}}
         assert_raises(mailer.MailerException,
                       mailer.mail_recipient, **test_email)
+
+    @helpers.change_config('smtp.reply_to', 'norply@ckan.org')
+    def test_reply_to(self):
+
+        msgs = self.get_smtp_messages()
+        assert_equal(msgs, [])
+
+        # send email
+        test_email = {'recipient_name': 'Bob',
+                      'recipient_email': 'Bob@bob.com',
+                      'subject': 'Meeting',
+                      'body': 'The meeting is cancelled.',
+                      'headers': {'header1': 'value1'}}
+        mailer.mail_recipient(**test_email)
+
+        # check it went to the mock smtp server
+        msgs = self.get_smtp_messages()
+        msg = msgs[0]
+
+        expected_from_header = "Reply-to: {}".format(
+            config.get('smtp.reply_to'))
+
+        assert_in(expected_from_header, msg[3])
Repository: ckan/ckan

Issue:
Missing dependency in requirements.txt (cookiecutter) 
https://github.com/ckan/ckan/blob/f2cea089bc0aaeede06d98449c4e9eb65e8c2f14/ckan/cli/generate.py#L7

- cookiecutter lib will be imported on a `ckan` cli attempt, but as it is missing from requirments.txt, is not present which will result to ImportError 

- cookiecutter is listed in requirments-dev.txt, but docker builds don't use it

Tested on a docker personal build, by : 
> docker build -t ckan .
> docker run --rm -it --entrypoint bash --name ckan -p 5000:5000 --link db:db --link redis:redis --link solr:solr ckan
> (activated-env)
> ckan


Discussion:

Resolving patch:
diff --git a/ckan/cli/generate.py b/ckan/cli/generate.py
--- a/ckan/cli/generate.py
+++ b/ckan/cli/generate.py
@@ -4,13 +4,21 @@
 import sys
 import click
 from ckan.cli import error_shout
-from cookiecutter.main import cookiecutter
 
 
-@click.group(name=u'generate',
-             short_help=u"Generate empty extension files to expand CKAN.")
+@click.group(
+    name=u'generate',
+    short_help=u"Generate empty extension files to expand CKAN.",
+    invoke_without_command=True,
+)
 def generate():
-    pass
+    try:
+        from cookiecutter.main import cookiecutter
+    except ImportError:
+        error_shout(u"`cookiecutter` library is missing from import path.")
+        error_shout(u"Make sure you have dev-dependencies installed:")
+        error_shout(u"\tpip install -r dev-requirements.txt")
+        raise click.Abort()
 
 
 @generate.command(name=u'extension', short_help=u"Create empty extension.")
@@ -18,6 +26,7 @@ def generate():
                                            u"template.",
               default=u'.')
 def extension(output_dir):
+    from cookiecutter.main import cookiecutter
     cur_loc = os.path.dirname(os.path.abspath(__file__))
     os.chdir(cur_loc)
     os.chdir(u'../../contrib/cookiecutter/ckan_extension/')


Test patch:
Repository: ckan/ckan

Issue:
Intermittent very high render times
### CKAN Version if known (or site URL)

2.7.3, 2.8.3

### Please describe the expected behaviour

Pages should normally load in under 10 seconds.

### Please describe the actual behaviour

Page loads are normally fast (around 1 second) but sometimes take over 30 seconds, causing proxies to time out. This is particularly problematic when it occurs during dataset creation, because it leaves the dataset in 'draft' state and cannot be resolved by resubmitting the request; the dataset must be located via the My Datasets page and edited to fix its state.

### What steps can be taken to reproduce the issue? 

No consistent cause has been found.


Discussion:
There's definitely more information needed here. There have been recent improvements in render times but the issues that they fixed would have consistent slow times, not spikes.
Spikes like these generally suggest a problem in the underlying infrastructure (db connections, memory issues etc).
If you can pinpoint some steps and conditions to consistently reproduce it feel free to reopen.
Just saw this happen in a development environment, so not under any special load, taking *85.044 seconds* to render `/dataset/new`. Tried it again and took 10.849 seconds.
Update: This has started happening consistently on new datasets in our production environment. Adding debugging statements indicates that the time is being spent in attempting to load tracking summary data (`ckan/model/tracking.py`, TrackingSummary.get_for_package), although in the end no tracking data exists since they're new datasets.
This appears to be related to the combination of ORDER BY and LIMIT 1 with a query that returns no results; see https://stackoverflow.com/questions/21385555/postgresql-query-very-slow-with-limit-1 and https://stackoverflow.com/questions/6037843/extremely-slow-postgresql-query-with-order-and-limit-clauses for discussion.

There is a simple workaround of adding a second unnecessary ordering clause, which causes PostgreSQL to make better query plan choices.

Resolving patch:
diff --git a/ckan/model/tracking.py b/ckan/model/tracking.py
--- a/ckan/model/tracking.py
+++ b/ckan/model/tracking.py
@@ -31,8 +31,8 @@ class TrackingSummary(domain_object.DomainObject):
     def get_for_package(cls, package_id):
         obj = meta.Session.query(cls).autoflush(False)
         obj = obj.filter_by(package_id=package_id)
-        data = obj.order_by(text('tracking_date desc')).first()
-        if data:
+        if meta.Session.query(obj.exists()).scalar():
+            data = obj.order_by(text('tracking_date desc')).first()
             return {'total' : data.running_total,
                     'recent': data.recent_views}
 


Test patch:
Repository: ckan/ckan

Issue:
Validation errors during resource creation lead to wrong URLs
This was hard to track as it doesn't happen in a standard CKAN install, as the default resource form doesn't have required fields.

If an extension implements custom validation in the resource schema, and there are errors returned in the form the File upload field will be set to the previously uploaded file name (ie a string).

When submitting, CKAN will interpret that like an external URL, and create a bogus `http://<file_name>` URL. There will be no errors returned so it's easier for users to assume that the file they first selected was successfully uploaded.

To reproduce:
1. Modify the resource schema, eg adding the `not_empty` validator to `description`
2. Go to the create resource page, select a file to upload, eg `libraries.csv`, leave the description empty and submit
3. You get redirected to the form page, with the description error, and also with "libraries.csv" set on the File field:
![Screenshot_2020-01-10 Add resource - Test URLs 4 - CKAN(1)](https://user-images.githubusercontent.com/200230/72160679-01314e00-33bf-11ea-89bb-dd04192a376d.png)

4. Submit the form. The resource is created, and has `http://libraries.csv` as URL.

The safest fix I can think of is resetting the upload field if there are validation errors. This will mean that users need to select the file again, but this is preferable to create a bad URL (we can not reset the field to the actual file previously selected because of browser security limitations). We can make this obvious to users with a message.



Discussion:

Resolving patch:
diff --git a/ckan/views/resource.py b/ckan/views/resource.py
--- a/ckan/views/resource.py
+++ b/ckan/views/resource.py
@@ -243,6 +243,10 @@ def post(self, package_type, id):
         except ValidationError as e:
             errors = e.error_dict
             error_summary = e.error_summary
+            if data.get(u'url_type') == u'upload' and data.get(u'url'):
+                data[u'url'] = u''
+                data[u'url_type'] = u''
+                data[u'previous_upload'] = True
             return self.get(package_type, id, data, errors, error_summary)
         except NotAuthorized:
             return base.abort(403, _(u'Unauthorized to create a resource'))


Test patch:
diff --git a/ckan/tests/controllers/test_user.py b/ckan/tests/controllers/test_user.py
--- a/ckan/tests/controllers/test_user.py
+++ b/ckan/tests/controllers/test_user.py
@@ -850,7 +850,7 @@ def test_request_reset_when_duplicate_emails(self, send_reset_link, app):
         emailed_users = [
             call[0][0].name for call in send_reset_link.call_args_list
         ]
-        assert emailed_users == [user_a["name"], user_b["name"]]
+        assert sorted(emailed_users) == sorted([user_a["name"], user_b["name"]])
 
     def test_request_reset_without_param(self, app):
Repository: ckan/ckan

Issue:
iDatastoreBackend tests doesn't get run
It looks like the tests for iDatastoreBackend don't get run: https://github.com/smotornyuk/ckan/blob/master/ckanext/example_idatastorebackend/test/test_plugin.py
because the classname doesn't start with `Test`.

There is [a clue](https://github.com/ckan/ckan/pull/3437/files#diff-075854f501da9e193409752e9156cddfR31) in the [original PR](https://github.com/ckan/ckan/pull/3437/files#diff-075854f501da9e193409752e9156cddfR28) from @smotornyuk that it was renamed from TestExampleIDatastoreBackendPlugin.

Perhaps this test can be re-enabled? (and modernized)

I was looking at this test because of querying a line in datastore/backend/postgres.py:_insert_links()


Discussion:

Resolving patch:
diff --git a/ckanext/example_idatastorebackend/example_sqlite.py b/ckanext/example_idatastorebackend/example_sqlite.py
--- a/ckanext/example_idatastorebackend/example_sqlite.py
+++ b/ckanext/example_idatastorebackend/example_sqlite.py
@@ -28,7 +28,7 @@ def _insert_records(self, table, records):
                         u', '.join(record.keys()),
                         u', '.join(['?'] * len(record.keys()))
                     ),
-                    record.values()
+                    list(record.values())
                 )
             pass
 


Test patch:
diff --git a/ckanext/example_idatastorebackend/test/test_plugin.py b/ckanext/example_idatastorebackend/test/test_plugin.py
--- a/ckanext/example_idatastorebackend/test/test_plugin.py
+++ b/ckanext/example_idatastorebackend/test/test_plugin.py
@@ -20,15 +20,10 @@
 )
 
 
-class ExampleIDatastoreBackendPlugin(helpers.FunctionalTestBase):
-    def setup(self):
-        super(ExampleIDatastoreBackendPlugin, self).setup()
-        plugins.load(u"datastore")
-        plugins.load(u"example_idatastorebackend")
-
-    def teardown(self):
-        plugins.unload(u"example_idatastorebackend")
-        plugins.unload(u"datastore")
+@pytest.mark.ckan_config(u"ckan.plugins",
+                         u"datastore example_idatastorebackend")
+@pytest.mark.usefixtures(u"with_plugins", u"clean_db", u"app")
+class TestExampleIDatastoreBackendPlugin():
 
     def test_backends_correctly_registered(self):
         DatastoreBackend.register_backends()
@@ -85,7 +80,7 @@ def test_backend_functionality(self, get_engine):
             records=records,
         )
         # check, create and 3 inserts
-        assert 5 == execute.call_count
+        assert 4 == execute.call_count
         insert_query = u'INSERT INTO "{0}"(a) VALUES(?)'.format(res["id"])
         execute.assert_has_calls(
             [
@@ -104,7 +99,7 @@ def test_backend_functionality(self, get_engine):
         fetchall.return_value = records
         helpers.call_action(u"datastore_search", resource_id=res["id"])
         execute.assert_called_with(
-            u'SELECT * FROM "{0}" LIMIT 10'.format(res["id"])
+            u'SELECT * FROM "{0}" LIMIT 100'.format(res["id"])
         )
 
         execute.reset_mock()
Repository: ckan/ckan

Issue:
Datapusher requests failing in 2.7.7 when using uwsgi sockets
### CKAN Version if known (or site URL)
2.7.7

### Please describe the expected behaviour
Datapusher requests to ckan should succeed when using nginx proxy and uwsgi sockets on the ckan server

### Please describe the actual behaviour
Datapusher fails with an error in the UI, and uswgi error in the ckan container logs: 
```
invalid request block size: 21327 (max 4096)...skip
```

### What steps can be taken to reproduce the issue? 
Run ckan with datapusher, ckan running uwsgi with flags:
```
--protocol uwsgi  --socket 0.0.0.0:5000
```

So this appears to have been introduced in https://github.com/ckan/ckan/commit/9a41f5a791bbc551959d3608103340ce545785c0

@amercader I'm confused as to why this url would be hardcoded like this? In our particular setup it almost works since we have our ckan container as a named route in the docker-compose file and we're running on port 5000, but since it bypasses the nginx proxy it is trying to talk http to uwsgi socket. Looking at the master branch it seems to be still in the same format as prior to your commit, `h.url_for('/', qualified=True)`.


Discussion:

Resolving patch:
diff --git a/ckanext/datapusher/logic/action.py b/ckanext/datapusher/logic/action.py
--- a/ckanext/datapusher/logic/action.py
+++ b/ckanext/datapusher/logic/action.py
@@ -60,7 +60,7 @@ def datapusher_submit(context, data_dict):
 
     datapusher_url = config.get('ckan.datapusher.url')
 
-    site_url = 'http://ckan:5000'
+    site_url = h.url_for('/', qualified=True)
 
     callback_url_base = config.get('ckan.datapusher.callback_url_base')
     if callback_url_base:


Test patch:
Repository: ckan/ckan

Issue:
Parsing dates with strict date formats
### Please describe the expected behaviour
When converting a date string to `datetime` , ckan accepts dates with either no decimal seconds or with full 6 decimal seconds as microseconds (e.g 30.374826 seconds). 
However, some generated date strings may contain more or less than 6 digits, like milliseconds with 3 digits, that we still want to use by converting them to the 6 digit microseconds.

### Please describe the actual behaviour
The `date_str_to_datetime` helper function only parses dates that have 6 decimal seconds (for microseconds) or non. It is failing when a date string has less or more than 6 decimal seconds, like 3 for milliseconds.

### What steps can be taken to reproduce the issue? 
In the `date_str_to_datetime` helper function, allow room to have more or less than 6 decimal seconds and then convert them into 6 digit microseconds.


Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1647,13 +1647,14 @@ def date_str_to_datetime(date_str):
 
     # Extract seconds and microseconds
     if len(time_tuple) >= 6:
-        m = re.match(r'(?P<seconds>\d{2})(\.(?P<microseconds>\d{6}))?$',
+        m = re.match(r'(?P<seconds>\d{2})(\.(?P<microseconds>\d+))?$',
                      time_tuple[5])
         if not m:
             raise ValueError('Unable to parse %s as seconds.microseconds' %
                              time_tuple[5])
         seconds = int(m.groupdict().get('seconds'))
-        microseconds = int(m.groupdict(0).get('microseconds'))
+        microseconds = int((str(m.groupdict(0).get('microseconds')) +
+                            '00000')[0:6])
         time_tuple = time_tuple[:5] + [seconds, microseconds]
 
     return datetime.datetime(*list(int(item) for item in time_tuple))


Test patch:
diff --git a/ckan/tests/legacy/lib/test_helpers.py b/ckan/tests/legacy/lib/test_helpers.py
--- a/ckan/tests/legacy/lib/test_helpers.py
+++ b/ckan/tests/legacy/lib/test_helpers.py
@@ -52,8 +52,8 @@ def test_date_str_to_datetime_with_garbage_on_end(self):
             h.date_str_to_datetime("2008-04-13T20:40:20foobar")
 
     def test_date_str_to_datetime_with_ambiguous_microseconds(self):
-        with pytest.raises(ValueError):
-            h.date_str_to_datetime("2008-04-13T20:40:20.500")
+        res = h.date_str_to_datetime("2008-04-13T20:40:20.1234")
+        assert res == datetime.datetime(2008, 4, 13, 20, 40, 20, 123400)
 
     def test_gravatar(self):
         email = "zephod@gmail.com"
Repository: ckan/ckan

Issue:
docs.ckan.org for 2.6, 2.7 and 2.8 haven't been updated since 2018
### Please describe the expected behaviour
https://docs.ckan.org/en/2.8/, https://docs.ckan.org/en/2.7/ and https://docs.ckan.org/en/2.6/ should have latest docs for each version.

### Please describe the actual behaviour
The docs are generated for 2.8.2, 2.7.5 and 2.6.7




Discussion:
Yes, unfortunately our old Jinja2 versions conflict with the ones used by RTD: https://github.com/readthedocs/readthedocs.org/issues/6894

The problem is that because of the way we generate the docstrings we need a full install of CKAN. At the end of the linked issue there is a suggested library that would avoid that.

Related: https://github.com/ckan/ckan/issues/4892

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -250,4 +250,8 @@ def parse_version(s):
         'Programming Language :: Python :: 2 :: Only',
         'Programming Language :: Python :: 2.7',
     ],
+    # this is used to fix an incompatiblity with readthedocs dependencies
+    extras_require={
+        "readthedocs":  ["Jinja2>=2.3"],
+    }
 )


Test patch:
Repository: ckan/ckan

Issue:
Make unflatten match flatten_dict > 1 level deep
### CKAN Version if known (or site URL)
all

### Please describe the expected behaviour
`unflatten(flatten_dict(x)) == x` for any `x` with more than 1 level of dict/list/dict nesting 

### Please describe the actual behaviour
unflatten returns dict with numeric keys past one level instead of lists

### What steps can be taken to reproduce the issue? 
see test in PR


Discussion:

Resolving patch:
diff --git a/ckan/lib/navl/dictization_functions.py b/ckan/lib/navl/dictization_functions.py
--- a/ckan/lib/navl/dictization_functions.py
+++ b/ckan/lib/navl/dictization_functions.py
@@ -410,24 +410,19 @@ def unflatten(data):
     for flattend_key in sorted(data.keys(), key=flattened_order_key):
         current_pos = unflattened
 
-        if (len(flattend_key) > 1
-                and not flattend_key[0] in convert_to_list
-                and not flattend_key[0] in unflattened):
-            convert_to_list.append(flattend_key[0])
-
         for key in flattend_key[:-1]:
             try:
                 current_pos = current_pos[key]
-            except KeyError:
+            except IndexError:
                 new_pos = {}
+                current_pos.append(new_pos)
+                current_pos = new_pos
+            except KeyError:
+                new_pos = []
                 current_pos[key] = new_pos
                 current_pos = new_pos
         current_pos[flattend_key[-1]] = data[flattend_key]
 
-    for key in convert_to_list:
-        unflattened[key] = [unflattened[key][s]
-                            for s in sorted(unflattened[key])]
-
     return unflattened
 
 


Test patch:
diff --git a/ckan/tests/legacy/lib/test_navl.py b/ckan/tests/legacy/lib/test_navl.py
--- a/ckan/tests/legacy/lib/test_navl.py
+++ b/ckan/tests/legacy/lib/test_navl.py
@@ -251,6 +251,27 @@ def test_flatten():
     assert data == unflatten(flatten_dict(data))
 
 
+def test_flatten_deeper():
+    data = {
+        u"resources": [
+            {
+                u"subfields": [
+                    {
+                        u"test": u"hello",
+                    },
+                ],
+            },
+        ],
+    }
+
+    assert flatten_dict(data) == {
+        ("resources", 0, u"subfields", 0, u"test"): u"hello",
+    }, pformat(flatten_dict(data))
+
+    assert data == unflatten(flatten_dict(data)), pformat(
+        unflatten(flatten_dict(data)))
+
+
 def test_simple():
     schema = {
         "name": [not_empty],