internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
Specify max resource size in an env variable
### CKAN Version if known (or site URL)
2.7.2 (applies to head of master as well)

### Please describe the expected behaviour
When deploying CKAN through docker it is not ideal to have to edit the configuration file in order to set maximum resource upload size. This would be better done through environment variables.

### Please describe the actual behaviour
Currently the ckan.max_upload_size config can only changed via editing the config .ini files.

### What steps can be taken to reproduce the issue? 
Running CKAN through Docker via Docker Compose.


Discussion:

Resolving patch:
diff --git a/ckan/config/environment.py b/ckan/config/environment.py
--- a/ckan/config/environment.py
+++ b/ckan/config/environment.py
@@ -143,7 +143,8 @@ def find_controller(self, controller):
     'smtp.starttls': 'CKAN_SMTP_STARTTLS',
     'smtp.user': 'CKAN_SMTP_USER',
     'smtp.password': 'CKAN_SMTP_PASSWORD',
-    'smtp.mail_from': 'CKAN_SMTP_MAIL_FROM'
+    'smtp.mail_from': 'CKAN_SMTP_MAIL_FROM',
+    'ckan.max_resource_size': 'CKAN_MAX_UPLOAD_SIZE_MB'
 }
 # End CONFIG_FROM_ENV_VARS
 


Test patch:
diff --git a/ckan/tests/config/test_environment.py b/ckan/tests/config/test_environment.py
--- a/ckan/tests/config/test_environment.py
+++ b/ckan/tests/config/test_environment.py
@@ -31,7 +31,8 @@ class TestUpdateConfig(h.FunctionalTestBase):
         ('CKAN_SMTP_STARTTLS', 'True'),
         ('CKAN_SMTP_USER', 'my_user'),
         ('CKAN_SMTP_PASSWORD', 'password'),
-        ('CKAN_SMTP_MAIL_FROM', 'server@example.com')
+        ('CKAN_SMTP_MAIL_FROM', 'server@example.com'),
+        ('CKAN_MAX_UPLOAD_SIZE_MB', '50')
     ]
 
     def _setup_env_vars(self):
@@ -71,6 +72,7 @@ def test_update_config_env_vars(self):
         nosetools.assert_equal(config['smtp.user'], 'my_user')
         nosetools.assert_equal(config['smtp.password'], 'password')
         nosetools.assert_equal(config['smtp.mail_from'], 'server@example.com')
+        nosetools.assert_equal(config['ckan.max_resource_size'], '50')
 
     def test_update_config_db_url_precedence(self):
         '''CKAN_SQLALCHEMY_URL in the env takes precedence over CKAN_DB'''
Repository: ckan/ckan

Issue:
Solr credential ENV vars not passed alongside CKAN_SOLR_URL
### CKAN Version if known (or site URL)

2.7.2

### Please describe the expected behaviour

Setting the following environment variables enables basic authentication for a Solr instance:

```
CKAN_SOLR_URL
CKAN_SOLR_USER
CKAN_SOLR_PASSWORD
```

### Please describe the actual behaviour

`CKAN_SOLR_URL` is honoured, but `CKAN_SOLR_USER` and `CKAN_SOLR_PASSWORD` are ignored and must be set in `ckan.ini`.

### What steps can be taken to reproduce the issue? 

1. Set the following environment variables:

```
CKAN_SOLR_USER=my_missing_user
CKAN_SOLR_PASSWORD=my_password
```

2. Restart CKAN
3. Observe that nothing has changed

Alternatively, create a Solr instance with Basic authentication, set the correct credentials as environment variables, then restart CKAN and watch the Solr connection fail.


Discussion:

Resolving patch:
diff --git a/ckan/config/environment.py b/ckan/config/environment.py
--- a/ckan/config/environment.py
+++ b/ckan/config/environment.py
@@ -135,6 +135,8 @@ def find_controller(self, controller):
     'ckan.datastore.read_url': 'CKAN_DATASTORE_READ_URL',
     'ckan.redis.url': 'CKAN_REDIS_URL',
     'solr_url': 'CKAN_SOLR_URL',
+    'solr_user': 'CKAN_SOLR_USER',
+    'solr_password': 'CKAN_SOLR_PASSWORD',
     'ckan.site_id': 'CKAN_SITE_ID',
     'ckan.site_url': 'CKAN_SITE_URL',
     'ckan.storage_path': 'CKAN_STORAGE_PATH',


Test patch:
Repository: ckan/ckan

Issue:
Datastore dump results are not the same as data in database
### CKAN Version if known (or site URL)
2.7.2

### Please describe the expected behaviour
Downloaded dataset is not consistent with the actual data! 

### Please describe the actual behaviour
We've upserted data using datastore api. The data is shown correctly in data explorer, Table view, API call or even querying postgres, But the data in downloaded csv file is different! Total number of records are the same but in csv we have random number of duplicates and missing data.

### What steps can be taken to reproduce the issue? 
Upsert data using API. Download the CSV file using URL. Filter and compare downloaded data and data shown in data explorer or table view.


Discussion:
Are you updating the data while dumping it by any chance? Dump is using row offsets which can change while data is being inserted.
It's highly unlikely. update is done once a week. I can provide a link if you want to check.
a link to the resource and examples of the duplicate+missing rows would help for sure.
Link to resource: https://data.illinois.gov/dataset/725state_employee_pay_hired_after_112011/resource/f86e27e4-67e4-4d50-9c52-a56c1159ace0
If you download the resource and filter the data using for example Agency = Human Rights you can see we have 145 records in CSV and we have 106 records in Data Explorer.

This does seem to be a pagination issue. We're requesting 25000 records at a time from the database and e.g. record `7301,...,2013,CORRECTIONS,SOUTHWESTERN IL..` appears at line 63899 and 88902 (25003 lines apart). We can force an ordering on `_id` here https://github.com/ckan/ckan/blob/master/ckanext/datastore/controller.py#L136

```python
    def result_page(offs, lim):
        return get_action('datastore_search')(None, {
            'resource_id': resource_id,
            'limit':
                PAGINATE_BY if limit is None
                else min(PAGINATE_BY, lim),
            'offset': offs,
            'sort': '_id',    # <----------------  add this line
            'records_format': records_format,
            'include_total': 'false',
        })
```
and we should stop seeing duplicates in the dump output
Thanks, 
I'll test it out and let you know.
I closed it for now since I cannot test it right now
Reopened because this one is serious. We need to fix & backport it even if you can't reproduce right now

Resolving patch:
diff --git a/ckanext/datastore/controller.py b/ckanext/datastore/controller.py
--- a/ckanext/datastore/controller.py
+++ b/ckanext/datastore/controller.py
@@ -137,6 +137,7 @@ def result_page(offs, lim):
                 PAGINATE_BY if limit is None
                 else min(PAGINATE_BY, lim),
             'offset': offs,
+            'sort': '_id',
             'records_format': records_format,
             'include_total': 'false',  # XXX: default() is broken
         })


Test patch:
Repository: ckan/ckan

Issue:
database system is starting up error
### CKAN Version if known (or site URL)
master

### Please describe the expected behaviour
when forcibly stopping postgres it can take longer to restart than normal. The CKAN process should wait for database while it is repairing.

### Please describe the actual behaviour
CKAN fails with a "database system is starting up" error message

### What steps can be taken to reproduce the issue? 
Not consistent, but use CKAN with docker and kill the postgres container then restart both postgres and ckan images.


Discussion:

Resolving patch:
diff --git a/ckan/model/__init__.py b/ckan/model/__init__.py
--- a/ckan/model/__init__.py
+++ b/ckan/model/__init__.py
@@ -4,6 +4,7 @@
 import logging
 import re
 from datetime import datetime
+from time import sleep
 
 from six import text_type
 import vdm.sqlalchemy
@@ -138,6 +139,7 @@
 log = logging.getLogger(__name__)
 
 
+DB_CONNECT_RETRIES = 10
 
 # set up in init_model after metadata is bound
 version_table = None
@@ -152,11 +154,18 @@ def init_model(engine):
     meta.metadata.bind = engine
     # sqlalchemy migrate version table
     import sqlalchemy.exc
-    try:
-        global version_table
-        version_table = Table('migrate_version', meta.metadata, autoload=True)
-    except sqlalchemy.exc.NoSuchTableError:
-        pass
+    for i in reversed(range(DB_CONNECT_RETRIES)):
+        try:
+            global version_table
+            version_table = Table('migrate_version', meta.metadata, autoload=True)
+            break
+        except sqlalchemy.exc.NoSuchTableError:
+            break
+        except sqlalchemy.exc.OperationalError as e:
+            if 'database system is starting up' in repr(e.orig) and i:
+                sleep(DB_CONNECT_RETRIES - i)
+                continue
+            raise
 
 
 class Repository(vdm.sqlalchemy.Repository):


Test patch:
Repository: ckan/ckan

Issue:
Upload logo is not working
### CKAN Version if known (or site URL)

2.8+ 
### Please describe the expected behaviour
When uploading logo from config page, we should see new logo on the portal

### Please describe the actual behaviour
Logo is not uploaded 

### What steps can be taken to reproduce the issue? 
Go to 
https://beta.ckan.org/ckan-admin/config

Upload an image 
Update config


Discussion:

Resolving patch:
diff --git a/ckan/views/admin.py b/ckan/views/admin.py
--- a/ckan/views/admin.py
+++ b/ckan/views/admin.py
@@ -110,11 +110,14 @@ def get(self):
 
     def post(self):
         try:
+            req = request.form.copy()
+            req.update(request.files.to_dict())
             data_dict = logic.clean_dict(
                 dict_fns.unflatten(
                     logic.tuplize_dict(
                         logic.parse_params(
-                            request.form, ignore_keys=CACHE_PARAMETERS))))
+                            req, ignore_keys=CACHE_PARAMETERS))))
+
             del data_dict['save']
             data = logic.get_action(u'config_option_update')({
                 u'user': g.user


Test patch:
Repository: ckan/ckan

Issue:
Internal server error when viewing /dashboard when logged out
### CKAN Version if known (or site URL)

2.8.0

### Please describe the expected behaviour

When attempting to visit /dashboard as a non-logged in user, the user should be sent to the login page.

### Please describe the actual behaviour

An internal server error occurs.

```
  File "/usr/lib/ckan/default/src/ckan/ckan/views/dashboard.py", line 99, in index
    u'id': g.userobj.id,
AttributeError: 'NoneType' object has no attribute 'id'
```

### What steps can be taken to reproduce the issue? 

Visit http://demo.ckan.org/dashboard when not logged in

Internal server error when viewing /dashboard when logged out
### CKAN Version if known (or site URL)

2.8.0

### Please describe the expected behaviour

When attempting to visit /dashboard as a non-logged in user, the user should be sent to the login page.

### Please describe the actual behaviour

An internal server error occurs.

```
  File "/usr/lib/ckan/default/src/ckan/ckan/views/dashboard.py", line 99, in index
    u'id': g.userobj.id,
AttributeError: 'NoneType' object has no attribute 'id'
```

### What steps can be taken to reproduce the issue? 

Visit http://demo.ckan.org/dashboard when not logged in



Discussion:

Resolving patch:
diff --git a/ckan/views/dashboard.py b/ckan/views/dashboard.py
--- a/ckan/views/dashboard.py
+++ b/ckan/views/dashboard.py
@@ -18,6 +18,9 @@
 @dashboard.before_request
 def before_request():
     try:
+        if not g.userobj:
+            raise logic.NotAuthorized()
+
         context = dict(model=model, user=g.user, auth_user_obj=g.userobj)
         logic.check_access(u'site_read', context)
     except logic.NotAuthorized:


Test patch:
diff --git a/ckan/tests/controllers/test_user.py b/ckan/tests/controllers/test_user.py
--- a/ckan/tests/controllers/test_user.py
+++ b/ckan/tests/controllers/test_user.py
@@ -192,6 +192,15 @@ def test_non_root_user_logout_url_redirect(self):
 
 class TestUser(helpers.FunctionalTestBase):
 
+    def test_not_logged_in_dashboard(self):
+        app = self._get_test_app()
+
+        for route in ['index', 'organizations', 'datasets', 'groups']:
+            app.get(
+                url=url_for(u'dashboard.{}'.format(route)),
+                status=403
+            )
+
     def test_own_datasets_show_up_on_user_dashboard(self):
         user = factories.User()
         dataset_title = 'My very own dataset'
Repository: ckan/ckan

Issue:
Remove authorization_group tables
### CKAN Version if known (or site URL)
Newest.

### Please describe the expected behaviour
Tables authorization_group, authorization_group_role, authorization_group_user  should not exist. There are no migrations to delete them.

### Please describe the actual behaviour
I've seen some work on cleaning authz: @rossjones working on it here https://github.com/ckan/ckan/commit/45c31815ea3282d185bde284788baf5da1b1749d, @tobes here https://github.com/ckan/ckan/commit/dc2ca3fe7185126cd455a7b96c7c72ca958411da, plus a PR https://github.com/ckan/ckan/pull/349.

I don't see migrations to delete those tables.

### What steps can be taken to reproduce the issue? 



Discussion:
Yes, these tables were used a long time ago. Feel free to submit a PR with a migration to get rid of them.

Resolving patch:
diff --git a/ckan/migration/versions/087_remove_old_authorization_tables.py b/ckan/migration/versions/087_remove_old_authorization_tables.py
new file mode 100644
--- /dev/null
+++ b/ckan/migration/versions/087_remove_old_authorization_tables.py
@@ -0,0 +1,10 @@
+# encoding: utf-8
+
+
+def upgrade(migrate_engine):
+    migrate_engine.execute(
+        '''
+            DROP TABLE IF EXISTS "authorization_group_user";
+            DROP TABLE IF EXISTS "authorization_group";
+        '''
+    )


Test patch:
Repository: ckan/ckan

Issue:
`search` legacy route missing
### CKAN Version if known (or site URL)
2.8+ (master)

### Please describe the expected behaviour
There is a shortcut for the `search` legacy route in CKAN 2.8+, I guess it's missing in [LEGACY_ROUTE_NAMES](https://github.com/ckan/ckan/blob/df35eec2f5229bc60176cf96dc83ca1fcc259d1e/ckan/lib/helpers.py#L68-L71).

### Please describe the actual behaviour
On master `search` is not available as route, basically the same as described in #4066

### What steps can be taken to reproduce the issue? 

```python
h.build_nav_main(
          ('home', _('Homepage')),
          ('search', _('Datasets')),
          ('group_index', _('Groups'))
        )
```


Discussion:
@metaodi it was not added as the 'package' controller was not merged in that time, but i think it could be added in the .ini file 
http://docs.ckan.org/en/2.8/maintaining/configuration.html#ckan-legacy-route-mappings

edit: but you are right, we should add it in the helper 
Well yes, but I think all legacy routes should be added there for compatibility reasons. If I provide a CKAN extension, it should work on all CKAN versions. The ini file might be a solution for tests, but I don't want all users of an extension to add `ckan.legacy_route_mappings` to their instances ini file, just because the extension runs on multiple versions of CKAN.

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -68,6 +68,8 @@
 LEGACY_ROUTE_NAMES = {
     'home': 'home.index',
     'about': 'home.about',
+    'search': 'dataset.search',
+    'organizations_index': 'organization.index'
 }
 
 


Test patch:
Repository: ckan/ckan

Issue:
i18n API returns JSON-encoded JSON
### CKAN Version if known (or site URL)
- Correct: https://demo.ckan.org/api/i18n/fi
- Incorrect: https://beta.ckan.org/api/i18n/fi

Problem probably stems from changes [here](https://github.com/ckan/ckan/blob/master/ckan/views/api.py#L467).

### Please describe the expected behaviour
The API should return JSON.

### Please describe the actual behaviour
The API returns JSON-encoded JSON. This breaks JavaScript translations, as the returned object deserializes into a string rather than an object.

### What steps can be taken to reproduce the issue? 
Call the API


Discussion:
[This](https://github.com/ckan/ckan/blob/master/ckan/views/api.py#L47) states `response_data` should be a string when `content_type` is not `text`, but [this](https://github.com/ckan/ckan/blob/master/ckan/views/api.py#L63) calls `json.dumps` on it if it's supposed to be JSON. The calling code conforms to the function documentation, but the function obviously expects a non-serialized value there.

Resolving patch:
diff --git a/ckan/views/api.py b/ckan/views/api.py
--- a/ckan/views/api.py
+++ b/ckan/views/api.py
@@ -44,7 +44,8 @@ def _finish(status_int, response_data=None,
     :param status_int: The HTTP status code to return
     :type status_int: int
     :param response_data: The body of the response
-    :type response_data: object if content_type is `text`, a string otherwise
+    :type response_data: object if content_type is `text` or `json`,
+        a string otherwise
     :param content_type: One of `text`, `html` or `json`. Defaults to `text`
     :type content_type: string
     :param headers: Extra headers to serve with the response
@@ -82,7 +83,8 @@ def _finish_ok(response_data=None,
     calling this method will prepare the response.
 
     :param response_data: The body of the response
-    :type response_data: object if content_type is `text`, a string otherwise
+    :type response_data: object if content_type is `text` or `json`,
+        a string otherwise
     :param content_type: One of `text`, `html` or `json`. Defaults to `json`
     :type content_type: string
     :param resource_location: Specify this if a new resource has just been
@@ -470,7 +472,7 @@ def i18n_js_translations(lang, ver=API_REST_DEFAULT_VERSION):
                              u'base', u'i18n', u'{0}.js'.format(lang)))
     if not os.path.exists(source):
         return u'{}'
-    translations = open(source, u'r').read()
+    translations = json.load(open(source, u'r'))
     return _finish_ok(translations)
 
 


Test patch:
Repository: ckan/ckan

Issue:
fq_list documentation
### CKAN Version if known (or site URL)
2.8.1

### Please describe the expected behaviour
`fq_list` can be set as value of  `data_dict` and it is used by [ckan/lib/search/query.py](https://github.com/ckan/ckan/blob/b9e45e2723d4abd70fa72b16ec4a0bebc795c56b/ckan/lib/search/query.py#L327).
'package_search` function in [ckan/logic/action/get.py](https://github.com/ckan/ckan/blob/b9e45e2723d4abd70fa72b16ec4a0bebc795c56b/ckan/logic/action/get.py#L1671) does not mention it in its docstring as possible param.

### Please describe the actual behaviour
`fq_list` should be documented. CKAN uses it only for its CLI at the moment, but other extensions could take advantage of it (I am using it to collapse datasets in combination with ckanext-datasetversions, in order to show just the most recent ones by default).


Discussion:

Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -1689,6 +1689,8 @@ def package_search(context, data_dict):
     :param fq: any filter queries to apply.  Note: ``+site_id:{ckan_site_id}``
         is added to this string prior to the query being executed.
     :type fq: string
+    :param fq_list: additional filter queries to apply.
+    :type fq_list: list of strings
     :param sort: sorting of the search results.  Optional.  Default:
         ``'relevance asc, metadata_modified desc'``.  As per the solr
         documentation, this is a comma-separated string of field names and


Test patch:
Repository: ckan/ckan

Issue:
url_validator fails for invalid URLs
### CKAN Version if known (or site URL)
CKAN 2.7, CKAN 2.8, master

### Please describe the expected behaviour
`url_validator` validates if a field contains a URL or not.

### Please describe the actual behaviour
The `url_validator` fails if certain invalid URLs are provided, e.g. `https://example.com]`.
The problem is, that `urlparse` raises a `ValueError` which is not caught in `url_validator`.

### What steps can be taken to reproduce the issue? 
1. Add the `url_validator` to a field
2. Provide the value `https://example.com]`
3. Server error with `ValueError: Invalid IPv6 URL` in logfile

I'm working on a PR to fix this issue.


Discussion:

Resolving patch:
diff --git a/ckan/logic/validators.py b/ckan/logic/validators.py
--- a/ckan/logic/validators.py
+++ b/ckan/logic/validators.py
@@ -690,18 +690,19 @@ def url_validator(key, data, errors, context):
     import urlparse
     import string
 
-    model = context['model']
-    session = context['session']
-
     url = data.get(key, None)
     if not url:
         return
 
-    pieces = urlparse.urlparse(url)
-    if all([pieces.scheme, pieces.netloc]) and \
-       set(pieces.netloc) <= set(string.letters + string.digits + '-.') and \
-       pieces.scheme in ['http', 'https']:
-       return
+    try:
+        pieces = urlparse.urlparse(url)
+        if all([pieces.scheme, pieces.netloc]) and \
+           set(pieces.netloc) <= set(string.letters + string.digits + '-.') and \
+           pieces.scheme in ['http', 'https']:
+           return
+    except ValueError:
+        # url is invalid
+        pass
 
     errors[key].append(_('Please provide a valid URL'))
 


Test patch:
diff --git a/ckan/tests/logic/test_validators.py b/ckan/tests/logic/test_validators.py
--- a/ckan/tests/logic/test_validators.py
+++ b/ckan/tests/logic/test_validators.py
@@ -611,4 +611,31 @@ def call_validator(*args, **kwargs):
         errors[key] = []
         call_validator(key, {key: password}, errors, None)
 
+
+class TestUrlValidator(object):
+
+    def test_ok(self):
+        urls = ['http://example.com', 'https://example.com', 'https://example.com/path?test=1&key=2']
+        key = ('url',)
+
+        @t.does_not_modify_errors_dict
+        def call_validator(*args, **kwargs):
+            return validators.url_validator(*args, **kwargs)
+        for url in urls:
+            errors = factories.validator_errors_dict()
+            errors[key] = []
+            call_validator(key, {key: url}, errors, None)
+
+    def test_invalid(self):
+        urls = ['ftp://example.com', 'test123', 'https://example.com]']
+        key = ('url',)
+
+        @adds_message_to_errors_dict('Please provide a valid URL')
+        def call_validator(*args, **kwargs):
+            return validators.url_validator(*args, **kwargs)
+        for url in urls:
+            errors = factories.validator_errors_dict()
+            errors[key] = []
+            call_validator(key, {key: url}, errors, None)
+
 # TODO: Need to test when you are not providing owner_org and the validator queries for the dataset with package_show