internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
Can not update user via `user_update` when using `name` for `id` parameter
**CKAN version**
2.9

**Describe the bug**
CKAN 2.9 introduced the `email_is_unique` validator to prevent duplicates. But the current implementation only works when the `id` param contains the actual UUID not the name, eg:

     http POST localhost:5000/api/action/user_update Authorization:{API_TOKEN} id=2e2dafdf-a7a8-495e-a9da-84ed4a1051a0 email=adria.mercader@okfn.org about=hola

Works fine, but this:

     http POST localhost:5000/api/action/user_update Authorization:{API_TOKEN} id=adria-mercader email=adria.mercader@okfn.org about=hola

Fails with a `ValidationError: "The email address 'adria.mercader@okfn.org' belongs to a registered user.`

The validator just needs to account for this scenario here:

https://github.com/ckan/ckan/blob/bcda3f6bfec5c68ac339070886702437df1c6ff2/ckan/logic/validators.py#L952-L953





Discussion:

Resolving patch:
diff --git a/ckan/logic/validators.py b/ckan/logic/validators.py
--- a/ckan/logic/validators.py
+++ b/ckan/logic/validators.py
@@ -936,20 +936,21 @@ def dict_only(value):
         raise Invalid(_('Must be a dict'))
     return value
 
+
 def email_is_unique(key, data, errors, context):
     '''Validate email is unique'''
     model = context['model']
     session = context['session']
 
     users = session.query(model.User) \
-            .filter(model.User.email == data[key]).all()
+        .filter(model.User.email == data[key]).all()
     # is there is no users with this email it's free
     if not users:
         return
     else:
         # allow user to update their own email
         for user in users:
-            if (user.name == data[("name",)]
+            if (user.name in [data[("name",)], data[("id",)]]
                     or user.id == data[("id",)]):
                 return
 


Test patch:
diff --git a/ckan/tests/logic/test_validators.py b/ckan/tests/logic/test_validators.py
--- a/ckan/tests/logic/test_validators.py
+++ b/ckan/tests/logic/test_validators.py
@@ -185,6 +185,22 @@ def test_email_is_unique_validator_user_update_email_unchanged(app):
         assert updated_user.email == old_email
 
 
+@pytest.mark.usefixtures("clean_db")
+def test_email_is_unique_validator_user_update_using_name_as_id(app):
+    with app.flask_app.test_request_context():
+        user = factories.User(username="user01", email="user01@email.com")
+
+        # try to update user1 and leave email unchanged
+        old_email = "user01@email.com"
+
+        helpers.call_action(
+            "user_update", id=user['name'], email=user['email'], about='test')
+        updated_user = model.User.get(user["id"])
+
+        assert updated_user.email == old_email
+        assert updated_user.about == 'test'
+
+
 @pytest.mark.usefixtures("clean_db")
 def test_email_is_unique_validator_user_update_email_new(app):
     with app.flask_app.test_request_context():
Repository: ckan/ckan

Issue:
Remove remaining Python 2 code in core
This should be done in separate pull requests to make reviews easier

- [x] Remove `requirements.py2.*` files and update documentation to remove py2 mentions
- [x] Remove py2 specific code. Look for `if six.PY2:` and remove what's inside!
- [x] Remove py3 checks. Look for `six.PY3` and make that the standard code run (remove the check if any)
- [x] Remove usage of six. We should not need the compatibility layer any more. Make all code standard Python 3

This should definitely be done separately

- [ ] Remove unicode literals (eg `my_conf[u"test_key_1"] = u"Test value 1"` -> `my_conf["test_key_1"] = "Test value 1"`



Discussion:
@amercader i'm working on this now.

For this `Remove requirements.py2.* files and update documentation to remove py2 mentions`: what part of the documentation do I need to update?
@amercader , I am interested to "Remove unicode literals". Where I need to remove it?
@steveoni my understanding is that you need to go look for any mentions of python 2 in the docs which is in the `/doc/` directory on this repository.
@Gauravp-NEC unicode literal need to be removed from the whole code base
@GabrielNicolasAvellaneda are you working on the removal of Unicode literals?
**Remove py3 checks. Look for six.PY3 and make that the standard code run (remove the check if any)**

Should test containing something like this`@pytest.mark.skipif(six.PY3, reason='')` be removed with all its content. e.g:

```js

@pytest.mark.skipif(six.PY3, reason='Only relevant to py2')
@mock.patch('paste.registry.TypeError')
def test_get_user_outside_web_request_py2(mock_TypeError):
    auth._get_user('example')
    assert mock_TypeError.called
```
@amercader 
@steveoni , yes, these are py2 only tests that no longer run

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -191,7 +191,7 @@
 
 extras_require = {}
 _extras_groups = [
-    ('requirements', 'requirements.txt'), ('requirements-py2', 'requirements-py2.txt'),
+    ('requirements', 'requirements.txt'),
     ('setuptools', 'requirement-setuptools.txt'), ('dev', 'dev-requirements.txt'),
 ]
 


Test patch:
Repository: ckan/ckan

Issue:
Can't use `solr_user` and `solr_password` in Python 3
**CKAN version**
2.9 

**Describe the bug**
When setting the `solr_user` and `solr_password` config options running Python 3 you get a startup error:

```
  File "/usr/lib/ckan/src/ckan/ckan/lib/search/__init__.py", line 309, in check_solr_schema_version
    res = _get_schema_from_solr(SOLR_SCHEMA_FILE_OFFSET_MANAGED)
  File "/usr/lib/ckan/src/ckan/ckan/lib/search/__init__.py", line 262, in _get_schema_from_solr
    http_auth = 'Basic ' + http_auth.encode('base64').strip()
LookupError: 'base64' is not a text encoding; use codecs.encode() to handle arbitrary codecs
```



Discussion:

Resolving patch:
diff --git a/ckan/lib/search/__init__.py b/ckan/lib/search/__init__.py
--- a/ckan/lib/search/__init__.py
+++ b/ckan/lib/search/__init__.py
@@ -5,6 +5,7 @@
 import sys
 import cgitb
 import warnings
+import base64
 import xml.dom.minidom
 
 import requests
@@ -259,7 +260,7 @@ def _get_schema_from_solr(file_offset):
     http_auth = None
     if solr_user is not None and solr_password is not None:
         http_auth = solr_user + ':' + solr_password
-        http_auth = 'Basic ' + http_auth.encode('base64').strip()
+        http_auth = 'Basic {}'.format(base64.b64encode(http_auth.encode('utf8')).strip())
 
     url = solr_url.strip('/') + file_offset
 


Test patch:
diff --git a/ckan/tests/lib/search/test_common.py b/ckan/tests/lib/search/test_common.py
new file mode 100644
--- /dev/null
+++ b/ckan/tests/lib/search/test_common.py
@@ -0,0 +1,12 @@
+# encoding: utf-8
+import pytest
+
+from ckan.common import config
+
+
+@pytest.mark.ckan_config("solr_user", "solr")
+@pytest.mark.ckan_config("solr_password", "password")
+def test_solr_user_and_password(app):
+
+    assert config["solr_user"] == "solr"
+    assert config["solr_password"] == "password"
Repository: ckan/ckan

Issue:
Search form validation
### CKAN Version if known (or site URL)
CKAN 2.8.3

### Please describe the expected behaviour
When user enter incorrect search query (in dataset search form on organization page), search form should inform her/him that something is wrong with that.

### Please describe the actual behaviour
Currently CKAN web UI returns "An internal server error occurs" message... which many users found to be misleading.

### What steps can be taken to reproduce the issue? 
Go into random organization page and enter "and" into search field and press enter, url should be similar to that:
http://<instance address>/organization/<organization name>?q=AND&sort=score+desc%2C+metadata_modified+desc


Discussion:
Since Pysolr returns us a pretty clear description of the exception that occurred, we can use it to explain to the user why exactly this happened.

`u"Solr responded with an error (HTTP 400): [Reason: Can't determine a Sort Order (asc or desc) in sort spec 'score dessc, metadata_modified descs', pos=5]".
`

For now, the message "There was an error while searching. Please try again." is hard-coded in the template, what is definitely could be unclear for someone. However, the text of the exception itself needs refinement before being presented to the user. I would like to work on this issues.

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1178,8 +1178,9 @@ def get_facet_items_dict(
     if search_facets is None:
         search_facets = getattr(c, u'search_facets', None)
 
-    if not search_facets or not search_facets.get(
-            facet, {}).get('items'):
+    if not search_facets \
+       or not isinstance(search_facets, dict) \
+       or not search_facets.get(facet, {}).get('items'):
         return []
     facets = []
     for facet_item in search_facets.get(facet)['items']:


Test patch:
Repository: ckan/ckan

Issue:
Inadequate null check in helpers.has_more_facets
**CKAN version**

2.8+

**Describe the bug**

The `has_more_facets` helper has an insufficient guard clause checking whether `c.search_facets_limits` is present, resulting in an attribute error if used without search_facets_limits present:

    if c.search_facets_limits and limit is None:
        limit = c.search_facets_limits.get(facet)

Contrast this with the guard in `get_facet_items_dict`:

    if hasattr(c, 'search_facets_limits'):
        if c.search_facets_limits and limit is None:
            limit = c.search_facets_limits.get(facet)

**Steps to reproduce**

Not sure how to recreate this in vanilla CKAN. We encountered it while doing development work with a mixture of `ckanext-datarequests` and `ckanext-s3filestore`.

**Expected behavior**

`c.search_facets_limits` should be ignored if not present.

**Additional details**

No stack trace occurs by default, merely an Internal Server Error page. Will submit a pull request soon.


Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1190,7 +1190,7 @@ def has_more_facets(facet, search_facets, limit=None, exclude_active=False):
             facets.append(dict(active=False, **facet_item))
         elif not exclude_active:
             facets.append(dict(active=True, **facet_item))
-    if c.search_facets_limits and limit is None:
+    if getattr(c, 'search_facets_limits', None) and limit is None:
         limit = c.search_facets_limits.get(facet)
     if limit is not None and len(facets) > limit:
         return True


Test patch:
Repository: ckan/ckan

Issue:
Add jinja2 loop controls extension and remove defunct with_ extension
**CKAN version**
master

**Describe the bug**
The jinja2 loop control extension adds `continue` and `break` statements in jinja templates.  The with_ extension is no longer needed as its built-in as of jinja2 2.9.

There's an existing PR for this - https://github.com/ckan/ckan/pull/5519



Discussion:

Resolving patch:
diff --git a/ckan/lib/jinja_extensions.py b/ckan/lib/jinja_extensions.py
--- a/ckan/lib/jinja_extensions.py
+++ b/ckan/lib/jinja_extensions.py
@@ -23,7 +23,7 @@
 
 
 def _get_extensions():
-    return ['jinja2.ext.do', 'jinja2.ext.with_',
+    return ['jinja2.ext.do', 'jinja2.ext.loopcontrols',
             SnippetExtension,
             CkanExtend,
             CkanInternationalizationExtension,


Test patch:
Repository: ckan/ckan

Issue:
Datapusher being triggered before file is uploaded (update)
**CKAN version**
v2.9, 2.8, 2.7

**Describe the bug**
The Datapusher is triggered before the file is uploaded and causing the file to use the old URL and the new data is not uploaded to filestore until clicked manually

**Steps to reproduce**
1. Create a resource
2. Check the preview
3. Update the resource with a new file.
4. The preview is same.
5. Click `Upload to Datapusher`. The resource will be updated

**Expected behavior**
The data from the new file should be updated to the datastore

**Additional details**

We can fix this by removing the `notify` method and using the `after_update` hook of the resource.

`notify` is triggered as soon the URL is changed and updated in DB and not committed to SOLR yet and the `resource_show` method use the SOLR to fetch the data so the datapusher is fetching from the old URL instead of new URL or a new resource



Discussion:
would this also affect xloader?
@duttonw I am not sure
@shubham-mahajan , I am interested to work on this issue. I have tried to reproduce this issue on CKAN 2.9.2 , but this issue is not reproduced in my environment. It is working as per expected behavior. 
@Gauravp-NEC Create a resource and upload a CSV, check the preview.

Now upload a completely different CSV and check the resource preview, it will not be updated until you manually click the `Upload the Datapusher`


@shubham-mahajan , I tried the same, but again this issue is not reproduced in my environment.

Resolving patch:
diff --git a/ckanext/datapusher/plugin.py b/ckanext/datapusher/plugin.py
--- a/ckanext/datapusher/plugin.py
+++ b/ckanext/datapusher/plugin.py
@@ -72,6 +72,10 @@ def after_resource_create(
 
         self._submit_to_datapusher(resource_dict)
 
+    def after_update(self, context, resource_dict):
+
+        self._submit_to_datapusher(resource_dict)
+
     def _submit_to_datapusher(self, resource_dict: dict[str, Any]):
         context = cast(Context, {
             u'model': model,


Test patch:
Repository: ckan/ckan

Issue:
jinja2.exceptions.UndefinedError: 'extras' is undefined
**CKAN version**
 master

**Describe the bug**
When trying to access `/dataset` i m getting this error: `jinja2.exceptions.UndefinedError: 'extras' is undefined`

![image](https://user-images.githubusercontent.com/72216462/134928400-f6cd4aab-9172-4233-a775-4abf6195a040.png)





Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1951,7 +1951,7 @@ def _create_url_with_params(
     params=None, controller=None, action=None, extras=None
 ):
     """internal function for building urls with parameters."""
-    if extras is None:
+    if not extras:
         if not controller and not action:
             # it's an url for the current page. Let's keep all interlal params,
             # like <package_type>


Test patch:
Repository: ckan/ckan

Issue:
CKAN 2.9.4 cannot load JS translations when using Python 2.7
This code is not Python 2.7 compatible:

https://github.com/ckan/ckan/blob/6731c5a821a6a5f4bdaa20f4e793e0b6ba44f823/ckan/views/api.py#L482

In Python 2, `encoding` is not a valid argument for the `open` function, but it is for the `json.load`.

Here is the commit which introduced the regression: https://github.com/ckan/ckan/commit/a280cf123748eb548a64242033aa41c0c67ee256

```python-traceback
2021-11-11 15:42:28,678 ERROR [ckan.config.middleware.flask_app] 'encoding' is an invalid keyword argument for this function
Traceback (most recent call last):
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1935, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib/ckan/venv/src/ckan/ckan/views/api.py", line 482, in i18n_js_translations
    translations = json.load(open(source, u'r', encoding=u'utf-8'))
TypeError: 'encoding' is an invalid keyword argument for this function
```

It seems to have been fixed partially already in another file: https://github.com/ckan/ckan/commit/b2b92a478f9a96c908e214d33d7d8141c73343c7

Related issue: #6051


Discussion:

Resolving patch:
diff --git a/ckan/views/api.py b/ckan/views/api.py
--- a/ckan/views/api.py
+++ b/ckan/views/api.py
@@ -3,6 +3,7 @@
 import os
 import logging
 import html
+import io
 
 from flask import Blueprint, make_response
 
@@ -449,7 +450,7 @@ def i18n_js_translations(lang, ver=API_REST_DEFAULT_VERSION):
                              u'base', u'i18n', u'{0}.js'.format(lang)))
     if not os.path.exists(source):
         return u'{}'
-    translations = json.load(open(source, u'r', encoding='utf-8'))
+    translations = json.load(io.open(source, u'r', encoding='utf-8'))
     return _finish_ok(translations)
 
 


Test patch:
Repository: ckan/ckan

Issue:
Non-root installs broken with python3
**CKAN version** 
2.9.3

**Describe the bug**
Root path is added twice to any generated URLs on the site. For example `/data/data/dataset`

**Steps to reproduce**
Install CKAN 2.9.3 into root_path `/data`, see `data-site-root` body tag attribute when loading `/data/dataset`

**Expected behavior**
The URLs are generated correctly

**Additional details**
This is because the Python 3 app stack does not contain any step to remove the `SCRIPT_NAME` environment variable set by the WSGI host. For Pylons this is done in RootPathMiddleware and in Python 2 installs `flask_app` does it in `can_handle_request`.

The issue can be fixed by adding a RootPathMiddleware inclusion step to `make_flask_stack`. I'll make a PR later today.



Discussion:

Resolving patch:
diff --git a/ckan/config/middleware/common_middleware.py b/ckan/config/middleware/common_middleware.py
--- a/ckan/config/middleware/common_middleware.py
+++ b/ckan/config/middleware/common_middleware.py
@@ -11,6 +11,24 @@
 from ckan.common import config
 
 
+class RootPathMiddleware(object):
+    '''
+    Prevents the SCRIPT_NAME server variable conflicting with the ckan.root_url
+    config. The routes package uses the SCRIPT_NAME variable and appends to the
+    path and ckan addes the root url causing a duplication of the root path.
+    This is a middleware to ensure that even redirects use this logic.
+    '''
+    def __init__(self, app, config):
+        self.app = app
+
+    def __call__(self, environ, start_response):
+        # Prevents the variable interfering with the root_path logic
+        if 'SCRIPT_NAME' in environ:
+            environ['SCRIPT_NAME'] = ''
+
+        return self.app(environ, start_response)
+
+
 class TrackingMiddleware(object):
 
     def __init__(self, app, config):
diff --git a/ckan/config/middleware/flask_app.py b/ckan/config/middleware/flask_app.py
--- a/ckan/config/middleware/flask_app.py
+++ b/ckan/config/middleware/flask_app.py
@@ -32,7 +32,8 @@
 from ckan.lib import i18n
 from ckan.common import config, g, request, ungettext
 from ckan.config.middleware.common_middleware import (TrackingMiddleware,
-                                                      HostHeaderMiddleware)
+                                                      HostHeaderMiddleware,
+                                                      RootPathMiddleware)
 import ckan.lib.app_globals as app_globals
 import ckan.lib.plugins as lib_plugins
 from ckan.lib.webassets_tools import get_webassets_path
@@ -205,6 +206,7 @@ def save_session(self, app, session, response):
         session_opts['session.data_dir'] = '{data_dir}/sessions'.format(
             data_dir=cache_dir)
 
+    app.wsgi_app = RootPathMiddleware(app.wsgi_app, session_opts)
     app.wsgi_app = SessionMiddleware(app.wsgi_app, session_opts)
     app.session_interface = BeakerSessionInterface()
 


Test patch:
Repository: ckan/ckan

Issue:
Upload.__init__ relies too much on makedirs
**CKAN version**

2.8, 2.9

**Describe the bug**

`ckan.lib.uploader:Upload.__init__` relies on `os.makedirs` to create the storage directory. However, this is inadequate if, for example, the storage directory has been created by another process, with sufficient permissions for CKAN to use it, yet with insufficient permissions in the parent directory for CKAN to have created it.

It also adds extra OS calls on every creation of an Upload object.



Discussion:
This code is used to create subdirectories for each uploader type, eg `groups` for group and org images, `admin` for the 
admin UI logos, and others that extensions might want to create, which is really useful.

https://github.com/ckan/ckan/blob/master/ckan/lib/uploader.py#L123-L130

Perhaps it's not the most elegant one, what do you propose to improve it? Maybe checking if the directories exist before attempting `os.makedirs`?
> Maybe checking if the directories exist before attempting os.makedirs?

It would be a good check to add, yes.

Consider a CKAN system created by a configuration management tool like Chef. It might well have the storage directory created, with appropriate permissions for CKAN to use it, but without write access to the parent directory.

Resolving patch:
diff --git a/ckan/lib/uploader.py b/ckan/lib/uploader.py
--- a/ckan/lib/uploader.py
+++ b/ckan/lib/uploader.py
@@ -122,12 +122,17 @@ def __init__(self, object_type, old_filename=None):
             return
         self.storage_path = os.path.join(path, 'storage',
                                          'uploads', object_type)
-        try:
-            os.makedirs(self.storage_path)
-        except OSError as e:
-            # errno 17 is file already exists
-            if e.errno != 17:
-                raise
+        # check if the storage directory is already created by
+        # the user or third-party
+        if os.path.isdir(self.storage_path):
+            pass
+        else:
+            try:
+                os.makedirs(self.storage_path)
+            except OSError as e:
+                # errno 17 is file already exists
+                if e.errno != 17:
+                    raise
         self.object_type = object_type
         self.old_filename = old_filename
         if old_filename:


Test patch:
Repository: ckan/ckan

Issue:
OneOf validator does not accept empty value
**CKAN version**
2.9, master

**Describe the bug**
OneOf validator was added in https://github.com/ckan/ckan/issues/4800, it is used in for example scheming where there are select fields where selecting one is not required. 2.8 accepts it as is, 2.9 fails in validation error. 

Simple test added to scheming here:

https://github.com/vrk-kpa/ckanext-scheming/blob/a4618a7c6dea180a125eaa52441ec9d1d528586a/ckanext/scheming/tests/test_validation.py#L71-L77

In 2.7 and 2.8 the test passes, on 2.9 it fails on validation error 

https://github.com/vrk-kpa/ckanext-scheming/runs/4548504508?check_suite_focus=true#step:7:431

![image](https://user-images.githubusercontent.com/830663/146389060-2ccb78d6-f2b0-4e30-87d3-a449fa7706fa.png)

This results in optional fields becoming required fields when upgrading ckan to 2.9.



Discussion:

Resolving patch:
diff --git a/ckan/logic/validators.py b/ckan/logic/validators.py
--- a/ckan/logic/validators.py
+++ b/ckan/logic/validators.py
@@ -951,9 +951,9 @@ def email_is_unique(key, data, errors, context):
 
 
 def one_of(list_of_value):
-    ''' Checks if the provided value is present in a list '''
+    ''' Checks if the provided value is present in a list or is an empty string'''
     def callable(value):
-        if value not in list_of_value:
+        if value != "" and value not in list_of_value:
             raise Invalid(_('Value must be one of {}'.format(list_of_value)))
         return value
     return callable


Test patch:
diff --git a/ckan/tests/logic/test_validators.py b/ckan/tests/logic/test_validators.py
--- a/ckan/tests/logic/test_validators.py
+++ b/ckan/tests/logic/test_validators.py
@@ -869,6 +869,11 @@ def test_val_not_in_list(self):
         func = validators.one_of(cont)
         raises_invalid(func)(5)
 
+    def test_empty_val_accepted(self):
+        cont = [1, 2, 3, 4]
+        func = validators.one_of(cont)
+        assert func("") == ""
+
 
 def test_tag_string_convert():
     def convert(tag_string):