internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
datastore set-up error - logging getting in the way
### CKAN Version if known (or site URL)
master

### Please describe the actual behaviour
Setting up the datastore permissions hits an error (which can be ignored, but still)
```
$ paster datastore set-permissions -c test-core.ini | sudo -u postgres psql
You are now connected to database "datastore_test" as user "postgres".
ERROR:  syntax error at or near "2017"
LINE 1: 2017-07-14 14:35:01,978 INFO  [ckan.lib.fanstatic_resources]...
        ^
```
This is because of some stray logging appearing:
```
$ paster datastore set-permissions -c test-core.ini
2017-07-14 14:37:10,623 INFO  [ckan.lib.fanstatic_resources] Base path /home/dread/ckan-master/ckan/ckan/public/base
/*
This script configures the permissions for the datastore.
...
```



Discussion:
Looks like it was added in this commit: https://github.com/ckan/ckan/commit/d03a88211990fe324ee73ebe3cacaf39e9ccda0f#diff-075debb789cb12033188527d800a4db5R239
I guess we can just lose the log statement.

Resolving patch:
diff --git a/ckan/lib/fanstatic_resources.py b/ckan/lib/fanstatic_resources.py
--- a/ckan/lib/fanstatic_resources.py
+++ b/ckan/lib/fanstatic_resources.py
@@ -235,7 +235,7 @@ def create_resource(path, lib_name, count, inline=False):
 base_path = os.path.abspath(os.path.join(
     os.path.dirname(__file__), '..', public, 'base'))
 
-log.info('Base path {0}'.format(base_path))
+log.debug('Base path {0}'.format(base_path))
 create_library('vendor', os.path.join(base_path, 'vendor'), depend_base=False)
 
 create_library('base', os.path.join(base_path, 'javascript'),
diff --git a/ckanext/datapusher/logic/action.py b/ckanext/datapusher/logic/action.py
--- a/ckanext/datapusher/logic/action.py
+++ b/ckanext/datapusher/logic/action.py
@@ -101,7 +101,7 @@ def datapusher_submit(context, data_dict):
                 # likely something went wrong with it and the state wasn't
                 # updated than its still in progress. Let it be restarted.
                 log.info('A pending task was found %r, but it is only %s hours'
-                         'old', existing_task['id'], time_since_last_updated)
+                         ' old', existing_task['id'], time_since_last_updated)
             else:
                 log.info('A pending task was found %s for this resource, so '
                          'skipping this duplicate task', existing_task['id'])


Test patch:
Repository: ckan/ckan

Issue:
Exception when specifying a directory in the ckan.i18n_directory option
### CKAN Version if known (or site URL)
2.6.2

### Please describe the expected behaviour
After specifying a directory in the `ckan.i18n_directory` option, we should get locales from that directory.

### Please describe the actual behaviour
Exception is thrown when accessing the CKAN site. The page shows "Internal Server Error".
```python
  File "/usr/lib/ckan/data_depositario/local/lib/python2.7/site-packages/pylons/controllers/core.py", line 60, in _perform_call
    return func(**args)
  File "/home/u10313335/ckan/lib/data_depositario/src/ckan/ckan/lib/base.py", line 200, in __before__
    i18n.handle_request(request, c)
  File "/home/u10313335/ckan/lib/data_depositario/src/ckan/ckan/lib/i18n.py", line 188, in handle_request
    set_lang(lang)
  File "/home/u10313335/ckan/lib/data_depositario/src/ckan/ckan/lib/i18n.py", line 238, in set_lang
    _set_lang(language_code)
  File "/home/u10313335/ckan/lib/data_depositario/src/ckan/ckan/lib/i18n.py", line 178, in _set_lang
    i18n.set_lang(lang, config=fake_config, class_=Translations)
  File "/usr/lib/ckan/data_depositario/local/lib/python2.7/site-packages/pylons/i18n/translation.py", line 182, in set_lang
    translator = _get_translator(lang, **kwargs)
  File "/usr/lib/ckan/data_depositario/local/lib/python2.7/site-packages/pylons/i18n/translation.py", line 168, in _get_translator
    languages=lang, **kwargs)
TypeError: translation() got an unexpected keyword argument 'config'
```

### What steps can be taken to reproduce the issue? 

1. Specify a directory with locales in the `ckan.i18n_directory` option. (Try the default `ckan/i18n` if you don't have one.)
2. Reload the CKAN site.


Discussion:
My workaround for this issue:

Change this line:

https://github.com/ckan/ckan/blob/a6fc30979dd75890c3322006554a350102f6b10b/ckan/lib/i18n.py#L178

from

```python
i18n.set_lang(lang, config=fake_config, class_=Translations)
```
to
```python
i18n.set_lang(lang, pylons_config=fake_config, class_=Translations)
```

Not sure if it is okay since we are migrating from pylons to flask.
I would submit a PR later If this fix looks good.
Hi @amercader ! 

I also saw this error when trying to upgrade to 2.6 
I'm translating CKAN and have the translations in my plugin.


Resolving patch:
diff --git a/ckan/lib/i18n.py b/ckan/lib/i18n.py
--- a/ckan/lib/i18n.py
+++ b/ckan/lib/i18n.py
@@ -225,7 +225,7 @@ def _set_lang(lang):
     if config.get('ckan.i18n_directory'):
         fake_config = {'pylons.paths': {'root': config['ckan.i18n_directory']},
                        'pylons.package': config['pylons.package']}
-        i18n.set_lang(lang, config=fake_config, class_=Translations)
+        i18n.set_lang(lang, pylons_config=fake_config, class_=Translations)
     else:
         i18n.set_lang(lang, class_=Translations)
 


Test patch:
Repository: ckan/ckan

Issue:
Registering a new account redirects to an unprefixed url
### CKAN Version if known (or site URL)
2.7.0 and master

### Please describe the expected behaviour
Upon registering a new account, the user should be redirected to a url starting with the root path

### Please describe the actual behaviour
The url to which a new user is redirected should include the root path

### What steps can be taken to reproduce the issue? 
On an installation that uses `ckan.root_path`, register a new account.


Discussion:

Resolving patch:
diff --git a/ckan/controllers/user.py b/ckan/controllers/user.py
--- a/ckan/controllers/user.py
+++ b/ckan/controllers/user.py
@@ -258,7 +258,7 @@ def _save_new(self, context):
         if not c.user:
             # log the user in programatically
             set_repoze_user(data_dict['name'])
-            h.redirect_to(controller='user', action='me', __ckan_no_root=True)
+            h.redirect_to(controller='user', action='me')
         else:
             # #1799 User has managed to register whilst logged in - warn user
             # they are not re-logged in as new user.


Test patch:
Repository: ckan/ckan

Issue:
Incorrect API link on /dataset
### CKAN Version if known (or site URL)
master

### Please describe the expected behaviour
Text at bottom of dataset listing (search.html template) should include a working API link

### Please describe the actual behaviour
A link of the form https://beta.ckan.org/packages?ver=%2F3 is generated
(https://beta.ckan.org/dataset)

This line is no longer working, I suspect because of the recent work to move the API over to Flask(?):
https://github.com/ckan/ckan/blob/master/ckan/templates/package/search.html#L52

### What steps can be taken to reproduce the issue? 

Incorrect API link on /dataset
### CKAN Version if known (or site URL)
master

### Please describe the expected behaviour
Text at bottom of dataset listing (search.html template) should include a working API link

### Please describe the actual behaviour
A link of the form https://beta.ckan.org/packages?ver=%2F3 is generated
(https://beta.ckan.org/dataset)

This line is no longer working, I suspect because of the recent work to move the API over to Flask(?):
https://github.com/ckan/ckan/blob/master/ckan/templates/package/search.html#L52

### What steps can be taken to reproduce the issue? 



Discussion:
Apparently the api link itself is also broken, which probably is caused by the same issue. For example https://beta.ckan.org/api/3 returns 404.
Apparently the api link itself is also broken, which probably is caused by the same issue. For example https://beta.ckan.org/api/3 returns 404.

Resolving patch:
diff --git a/ckan/config/middleware/flask_app.py b/ckan/config/middleware/flask_app.py
--- a/ckan/config/middleware/flask_app.py
+++ b/ckan/config/middleware/flask_app.py
@@ -355,7 +355,9 @@ def _register_core_blueprints(app):
     def is_blueprint(mm):
         return isinstance(mm, Blueprint)
 
-    for loader, name, _ in pkgutil.iter_modules(['ckan/views'], 'ckan.views.'):
+    path = os.path.join(os.path.dirname(__file__), '..', '..', 'views')
+
+    for loader, name, _ in pkgutil.iter_modules([path], 'ckan.views.'):
         module = loader.find_module(name).load_module(name)
         for blueprint in inspect.getmembers(module, is_blueprint):
             app.register_blueprint(blueprint[1])


Test patch:
Repository: ckan/ckan

Issue:
Permission labels are indexed by type text in SOLR
### CKAN Version if known (or site URL)
2.7, master

### Please describe the expected behaviour
Organization Admin should see organizations private datasets even when SOLR indexing has been changed to accommodate different languages. The permission label should not be index by language rules.

### Please describe the actual behaviour
If indexing rules are changed for example to use [EdgeNGram](https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.EdgeNGramFilterFactory), the permission label id's are indexed by these rules and organization admin, editor or member can not see organizations private datasets.


### What steps can be taken to reproduce the issue? 
Apply EdgeNGram to solr indexing.

Create organization.
Create user.
Add user to organization.
Create private Dataset to organization.

### Possible fix
Use type textgen instead of text in the default schema.xml.



Discussion:
@Zharktas sounds right to me. Mind submitting a PR?
Use `string` not `textgen`, which in CKAN's schema is not the same as the old Solr 1.* `textgen` FieldType (confusing I know).
@wardi submitted minor PR for this.

Resolving patch:
diff --git a/ckan/lib/search/__init__.py b/ckan/lib/search/__init__.py
--- a/ckan/lib/search/__init__.py
+++ b/ckan/lib/search/__init__.py
@@ -31,7 +31,7 @@ def text_traceback():
     return res
 
 
-SUPPORTED_SCHEMA_VERSIONS = ['2.7']
+SUPPORTED_SCHEMA_VERSIONS = ['2.8']
 
 DEFAULT_OPTIONS = {
     'limit': 20,


Test patch:
Repository: ckan/ckan

Issue:
Change CORS header keys and values to string instead of unicode
The CORS header keys and values being unicode raised problems when using uWSGI with CKAN. uWSGI expects the header keys to be string instead of unicode which results in a TypeError. 

### Features:

- [ ] includes tests covering changes
- [ ] includes updated documentation
- [ ] includes user-visible changes
- [ ] includes API changes
- [x] includes bugfix for possible backport

Please [X] all the boxes above that apply



Discussion:
This makes our test that checks for string literals fail. Rather than remove the `u''` bit, can you replace it by a `b''`? Check the docs on this: http://docs.ckan.org/en/ckan-2.7.2/contributing/unicode.html#string-literals

cc @torfsen 
Thanks @ljupchokotev!

Resolving patch:
diff --git a/ckan/views/__init__.py b/ckan/views/__init__.py
--- a/ckan/views/__init__.py
+++ b/ckan/views/__init__.py
@@ -54,7 +54,7 @@ def set_cors_headers_for_response(response):
 
         cors_origin_allowed = None
         if asbool(config.get(u'ckan.cors.origin_allow_all')):
-            cors_origin_allowed = u'*'
+            cors_origin_allowed = b'*'
         elif config.get(u'ckan.cors.origin_whitelist') and \
                 request.headers.get(u'Origin') \
                 in config[u'ckan.cors.origin_whitelist'].split(u' '):


Test patch:
Repository: ckan/ckan

Issue:
Paster/CLI config-tool requires _get_test_app which in turn requires a dev-only dependency
### CKAN Version if known (or site URL)
2.7/close-to-master

### Please describe the expected behaviour
Paster can be used

### Please describe the actual behaviour
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 31, in <module>
from ckan.tests.helpers import _get_test_app
File "/usr/lib/ckan/default/src/ckan/ckan/tests/helpers.py", line 34, in <module>
import mock
ImportError: No module named mock

### What steps can be taken to reproduce the issue?
Do not pip install dev-requirements and then use paster config-tool (`paster --plugin=ckan config-tool`)



Discussion:
Tried adding ```dev-requirements.txt``` to the ```Dockerfile``` to get mock, but still getting the same error. Messages from ```docker-compose logs -f ckan``` do report successful install of mock.

```
...
RUN ckan-pip install --upgrade -r $CKAN_VENV/src/ckan/requirements.txt && \
    ckan-pip install --upgrade -r $CKAN_VENV/src/ckan/dev-requirements.txt && \
    ckan-pip install -e $CKAN_VENV/src/ckan/ && \
...
```
@AnveshC Not yet. I haven't had enough time to really dig into the issue.
it seems to be introduced by https://github.com/ckan/ckan/pull/3782 as a conscious tradeoff, so in line with that it might be most appropriate to move the mock dependency from `dev-requirements.txt` to the `requirements.txt` file.

@goatsweater it still not picking up on mock even though it's installs when building I think is caused by the `ckan_home` volume. For me `docker-compose down`, `docker volume prune` and `docker-compose up` resolved that problem.

Resolving patch:
diff --git a/ckan/lib/cli.py b/ckan/lib/cli.py
--- a/ckan/lib/cli.py
+++ b/ckan/lib/cli.py
@@ -22,13 +22,13 @@
 from paste.script.util.logging_config import fileConfig
 import click
 
+from ckan.config.middleware import make_app
 import ckan.logic as logic
 import ckan.model as model
 import ckan.include.rjsmin as rjsmin
 import ckan.include.rcssmin as rcssmin
 import ckan.plugins as p
 from ckan.common import config
-from ckan.tests.helpers import _get_test_app
 
 # This is a test Flask request context to be used internally.
 # Do not use it!
@@ -232,7 +232,9 @@ def load_config(config, load_site_user=True):
     # Set this internal test request context with the configured environment so
     # it can be used when calling url_for from the CLI.
     global _cli_test_request_context
-    flask_app = _get_test_app().flask_app
+
+    app = make_app(conf.global_conf, **conf.local_conf)
+    flask_app = app.apps['flask_app']._wsgi_app
     _cli_test_request_context = flask_app.test_request_context()
 
     registry = Registry()


Test patch:
Repository: ckan/ckan

Issue:
Package resources not included when installing from source in non-editable mode.
Known affects:
 - CKAN 2.7.2; Python 2.7.14; Ubuntu 16.04 & 17.10; 
   ```
   $ virtualenv --version
   15.1.0
   $ virtualenv --no-site-packages test-venv
   <snip>
   $ ./test-venv/bin/pip freeze --all 
   pip==9.0.1
   setuptools==38.2.4
   wheel==0.30.0
    ```

### Expected behaviour
Checking out the repository and installing it(without the editable flag) should install the required package resources like `ckan/migration/migrate.cfg`.

## Actual behaviour
Installing the package without the editable flag does not include the package resources meaning all JavaScript, CSS, templates and the migration config noted above are not included.

This makes the package non-functional. 

For me the problem arose because `ckan/migration/migrate.cfg` did not exist in my install directory, and hence the database could not be created. There are numerous other files listed in `MANIFEST.in` which are also required for CKAN to run which are not included.

### What steps can be taken to reproduce the issue? 

Following the [installing from source](http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html) instructions, omitting the `-e` flag from step **c**

## Resolution.

The solution to this is to add `include_package_data=True` to the `setup` call in `setup.py`




Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -216,6 +216,7 @@ def parse_version(s):
     keywords='data packaging component tool server',
     long_description=__long_description__,
     zip_safe=False,
+    include_package_data=True,
     packages=find_packages(exclude=['ez_setup']),
     namespace_packages=['ckanext', 'ckanext.stats'],
     message_extractors={


Test patch:
Repository: ckan/ckan

Issue:
Add support for postgresql 10
Updated the psycopg2 drivers to support latest Postgresql version

Fixes #

### Proposed fixes:



### Features:

- [x] includes tests covering changes
- [ ] includes updated documentation
- [ ] includes user-visible changes
- [ ] includes API changes
- [ ] includes bugfix for possible backport

Please [X] all the boxes above that apply



Discussion:
@groundrace  thanks for this. We need to ensure that the actual code also works against postgres 10. Looking at the failures it looks like there are some JSON related ones on the DataStore, so perhaps this needs to be investigated a bit more. Can you check what needs updating?

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -23,7 +23,7 @@
 import ckanext.datastore.helpers as datastore_helpers
 import ckanext.datastore.interfaces as interfaces
 
-import psycopg2.extras
+from psycopg2.extras import register_default_json, register_composite
 import distutils.version
 from sqlalchemy.exc import (ProgrammingError, IntegrityError,
                             DBAPIError, DataError)
@@ -117,6 +117,14 @@ def _get_engine_from_url(connection_url):
                                                'ckan.datastore.sqlalchemy.',
                                                **extras)
         _engines[connection_url] = engine
+
+    # don't automatically convert to python objects
+    # when using native json types in 9.2+
+    # http://initd.org/psycopg/docs/extras.html#adapt-json
+    register_default_json(conn_or_curs=engine.raw_connection().connection,
+                          globally=False,
+                          loads=lambda x: x)
+
     return engine
 
 
@@ -274,9 +282,7 @@ def _cache_types(context):
             # redo cache types with json now available.
             return _cache_types(context)
 
-        psycopg2.extras.register_composite('nested',
-                                           connection.connection,
-                                           True)
+        register_composite('nested', connection.connection, True)
 
 
 def _pg_version_is_at_least(connection, version):


Test patch:
diff --git a/ckanext/datastore/tests/test_unit.py b/ckanext/datastore/tests/test_unit.py
--- a/ckanext/datastore/tests/test_unit.py
+++ b/ckanext/datastore/tests/test_unit.py
@@ -43,7 +43,7 @@ def test_pg_version_check(self):
         engine = db._get_engine_from_url(config['sqlalchemy.url'])
         connection = engine.connect()
         assert db._pg_version_is_at_least(connection, '8.0')
-        assert not db._pg_version_is_at_least(connection, '10.0')
+        assert not db._pg_version_is_at_least(connection, '20.0')
 
 
 class TestLegacyModeSetting():
Repository: ckan/ckan

Issue:
making a JSONP call to the CKAN API returns the wrong mime type
### CKAN Version if known (or site URL)
2.7

### Please describe the expected behaviour
When making an api call to the CKAN api, and using the callback parameter to generate jsonp instead of json, the content-type should be set to "application/javascript"

### Please describe the actual behaviour
the content-type is set to application/json regardless of whether the callback parameter is used or not

### What steps can be taken to reproduce the issue? 
Step 1. Make a request to https://demo.ckan.org/api/3/action/package_search?q=spending&callback=myFunc
Step 2. Inspect the response headers, and notice that the Content-Type header is set to application/json



Discussion:

Resolving patch:
diff --git a/ckan/controllers/api.py b/ckan/controllers/api.py
--- a/ckan/controllers/api.py
+++ b/ckan/controllers/api.py
@@ -38,6 +38,7 @@
     'text': 'text/plain;charset=utf-8',
     'html': 'text/html;charset=utf-8',
     'json': 'application/json;charset=utf-8',
+    'javascript': 'application/javascript;charset=utf-8',
 }
 
 
@@ -99,6 +100,7 @@ def _finish(self, status_int, response_data=None,
                 # escape callback to remove '<', '&', '>' chars
                 callback = cgi.escape(request.params['callback'])
                 response_msg = self._wrap_jsonp(callback, response_msg)
+                response.headers['Content-Type'] = CONTENT_TYPES['javascript']
         return response_msg
 
     def _finish_ok(self, response_data=None,
diff --git a/ckan/views/api.py b/ckan/views/api.py
--- a/ckan/views/api.py
+++ b/ckan/views/api.py
@@ -23,6 +23,7 @@
     u'text': u'text/plain;charset=utf-8',
     u'html': u'text/html;charset=utf-8',
     u'json': u'application/json;charset=utf-8',
+    u'javascript': u'application/javascript;charset=utf-8',
 }
 
 API_REST_DEFAULT_VERSION = 1
@@ -69,6 +70,7 @@ def _finish(status_int, response_data=None,
             # escape callback to remove '<', '&', '>' chars
             callback = cgi.escape(request.args[u'callback'])
             response_msg = _wrap_jsonp(callback, response_msg)
+            headers[u'Content-Type'] = CONTENT_TYPES[u'javascript']
     return make_response((response_msg, status_int, headers))
 
 


Test patch:
diff --git a/ckan/tests/controllers/test_api.py b/ckan/tests/controllers/test_api.py
--- a/ckan/tests/controllers/test_api.py
+++ b/ckan/tests/controllers/test_api.py
@@ -278,6 +278,19 @@ def test_jsonp_works_on_get_requests(self):
         eq_(sorted(res_dict['result']),
             sorted([dataset1['name'], dataset2['name']]))
 
+    def test_jsonp_returns_javascript_content_type(self):
+        url = url_for(
+            controller='api',
+            action='action',
+            logic_function='status_show',
+            ver='/3')
+        app = self._get_test_app()
+        res = app.get(
+            url=url,
+            params={'callback': 'my_callback'},
+        )
+        assert_in('application/javascript', res.headers.get('Content-Type'))
+
     def test_jsonp_does_not_work_on_post_requests(self):
 
         dataset1 = factories.Dataset()
Repository: ckan/ckan

Issue:
Exception in search page when not authorized
If someone customized the `package_search` auth function to restrict access the frontend returns an uncaught exception as this case is not handled in the controller


Discussion:

Resolving patch:
diff --git a/ckan/controllers/package.py b/ckan/controllers/package.py
--- a/ckan/controllers/package.py
+++ b/ckan/controllers/package.py
@@ -311,6 +311,9 @@ def pager_url(q=None, page=None):
             c.query_error = True
             c.search_facets = {}
             c.page = h.Page(collection=[])
+        except NotAuthorized:
+            abort(403, _('Not authorized to see this page'))
+
         c.search_facets_limits = {}
         for facet in c.search_facets.keys():
             try:


Test patch: