internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
Provide a mechanism to support CSRF on extensions that support older CKAN version
@TomeCirun @pdelboca 

After #6863, [best practices](https://github.com/ckan/ckan/blob/master/doc/extensions/best-practices.rst) say:

> To add CSRF protection to your extensions use the CSRF macro in all your forms

> {% import 'macros/form.html' as form %}
> ...
> {{ form.csrf_protection() }}

This won't work on CKAN<=2.9 because that macro doesn't exist. Similarly, trying to add the input directly like in [here](https://github.com/ckan/ckan/pull/6863/files#diff-09d2291ff01f465bfadf71b865872b83bd0301fc32f72e3fc96964a74a1619b0R7) will fail because the `csrf_token()` function is not defined.

My suggestion would be:

1. Consolidate all CSRF inputs into a single `{{ h.csrf_input() }}` call (both the ` {{ form.csrf_protection() }}` ones and the direct ` <input type="hidden" ...` ones). Using a helper will allow extensions to override it if necessary. Then we can use the same template markup everywhere.
2. Provide `h.csrf_input()` stubs on CKAN<2.9 that don't actually return anything. This could be done either:
   2.1. Patching CKAN core so 2.9.6 and 2.8.11 register `csrf_input()` template helpers, but they return an empty string
   2.2 Something similar but using ckantoolkit. I think this would be messier and even impossible.

At the end of this, an extension wanting to support CKAN<=2.9 and CKAN 2.10 with CSRF protection can just add the same code as core, and it will work across CKAN versions without messy checks on each extension:

```
  <form class="dataset-form add-member-form" method='post'>
    {{ h.csrf_input() }}
```

Does this make sense?

**edit**: updated to use a template helper instead of a Jinja tag


Discussion:
Forgot to say that this should be validated on everyone's favourite forms extension, ckanext-scheming!
@TomeCirun Does the above make sense? would you have time to work on it?
Yes @amercader, I can work on this

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -2783,3 +2783,8 @@ def make_login_url(
         parsed_base = parsed_base._replace(netloc=netloc, query=urlencode(md))
         return urlunparse(parsed_base)
     return base
+
+
+@core_helper
+def csrf_input():
+    return snippet('snippets/csrf_input.html')
diff --git a/ckan/plugins/toolkit.py b/ckan/plugins/toolkit.py
--- a/ckan/plugins/toolkit.py
+++ b/ckan/plugins/toolkit.py
@@ -47,7 +47,7 @@
     literal,
     chained_helper,
     redirect_to,
-    url_for,
+    url_for
 )
 from ckan.exceptions import (
     CkanVersionException,
@@ -99,7 +99,7 @@
     "mail_recipient", "mail_user",
     "render_snippet", "add_template_directory", "add_public_directory",
     "add_resource", "add_ckan_admin_tab",
-    "check_ckan_version", "requires_ckan_version", "get_endpoint",
+    "check_ckan_version", "requires_ckan_version", "get_endpoint"
 ]
 
 get_converter = get_validator


Test patch:
Repository: ckan/ckan

Issue:
Provide a mechanism to support CSRF on extensions that support older CKAN version
@TomeCirun @pdelboca 

After #6863, [best practices](https://github.com/ckan/ckan/blob/master/doc/extensions/best-practices.rst) say:

> To add CSRF protection to your extensions use the CSRF macro in all your forms

> {% import 'macros/form.html' as form %}
> ...
> {{ form.csrf_protection() }}

This won't work on CKAN<=2.9 because that macro doesn't exist. Similarly, trying to add the input directly like in [here](https://github.com/ckan/ckan/pull/6863/files#diff-09d2291ff01f465bfadf71b865872b83bd0301fc32f72e3fc96964a74a1619b0R7) will fail because the `csrf_token()` function is not defined.

My suggestion would be:

1. Consolidate all CSRF inputs into a single `{{ h.csrf_input() }}` call (both the ` {{ form.csrf_protection() }}` ones and the direct ` <input type="hidden" ...` ones). Using a helper will allow extensions to override it if necessary. Then we can use the same template markup everywhere.
2. Provide `h.csrf_input()` stubs on CKAN<2.9 that don't actually return anything. This could be done either:
   2.1. Patching CKAN core so 2.9.6 and 2.8.11 register `csrf_input()` template helpers, but they return an empty string
   2.2 Something similar but using ckantoolkit. I think this would be messier and even impossible.

At the end of this, an extension wanting to support CKAN<=2.9 and CKAN 2.10 with CSRF protection can just add the same code as core, and it will work across CKAN versions without messy checks on each extension:

```
  <form class="dataset-form add-member-form" method='post'>
    {{ h.csrf_input() }}
```

Does this make sense?

**edit**: updated to use a template helper instead of a Jinja tag


Discussion:
Forgot to say that this should be validated on everyone's favourite forms extension, ckanext-scheming!
@TomeCirun Does the above make sense? would you have time to work on it?
Yes @amercader, I can work on this

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -2703,3 +2703,8 @@ def sanitize_id(id_):
     ValueError.
     '''
     return str(uuid.UUID(id_))
+
+
+@core_helper
+def csrf_input():
+    return ''


Test patch:
Repository: ckan/ckan

Issue:
Provide a mechanism to support CSRF on extensions that support older CKAN version
@TomeCirun @pdelboca 

After #6863, [best practices](https://github.com/ckan/ckan/blob/master/doc/extensions/best-practices.rst) say:

> To add CSRF protection to your extensions use the CSRF macro in all your forms

> {% import 'macros/form.html' as form %}
> ...
> {{ form.csrf_protection() }}

This won't work on CKAN<=2.9 because that macro doesn't exist. Similarly, trying to add the input directly like in [here](https://github.com/ckan/ckan/pull/6863/files#diff-09d2291ff01f465bfadf71b865872b83bd0301fc32f72e3fc96964a74a1619b0R7) will fail because the `csrf_token()` function is not defined.

My suggestion would be:

1. Consolidate all CSRF inputs into a single `{{ h.csrf_input() }}` call (both the ` {{ form.csrf_protection() }}` ones and the direct ` <input type="hidden" ...` ones). Using a helper will allow extensions to override it if necessary. Then we can use the same template markup everywhere.
2. Provide `h.csrf_input()` stubs on CKAN<2.9 that don't actually return anything. This could be done either:
   2.1. Patching CKAN core so 2.9.6 and 2.8.11 register `csrf_input()` template helpers, but they return an empty string
   2.2 Something similar but using ckantoolkit. I think this would be messier and even impossible.

At the end of this, an extension wanting to support CKAN<=2.9 and CKAN 2.10 with CSRF protection can just add the same code as core, and it will work across CKAN versions without messy checks on each extension:

```
  <form class="dataset-form add-member-form" method='post'>
    {{ h.csrf_input() }}
```

Does this make sense?

**edit**: updated to use a template helper instead of a Jinja tag


Discussion:
Forgot to say that this should be validated on everyone's favourite forms extension, ckanext-scheming!
@TomeCirun Does the above make sense? would you have time to work on it?
Yes @amercader, I can work on this

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -2987,3 +2987,8 @@ def can_update_owner_org(package_dict, user_orgs=None):
         return True
 
     return False
+
+
+@core_helper
+def csrf_input():
+    return ''


Test patch:
Repository: ckan/ckan

Issue:
IConfigurer plugin load order
**CKAN version**
(all)

**Describe the bug**
`update_config` runs through all IConfigurer plugins from first to last calling `plugin.update_config`. The pattern for other interfaces is that the "first plugin wins", but this is difficult to implement when later plugins override values from earlier ones in the list.

**Steps to reproduce**
Enable two plugins that set the same config value using IConfigurer

**Expected behavior**
First plugin value should win, like with other interfaces.

**Additional details**
ckanext-envvars recommends adding `envvars` last in the list of plugins, which makes sense but if other plugins depend on/override values configured in envvars (e.g. ckanext-scheming) they won't be available at `update_config` time.


Discussion:
We decided to [reverse the order](https://github.com/ckan/ckan/blob/f2eb20ebf9de21016fd3d15ed028be1cce8d447c/ckan/config/environment.py#L118) in which `IConfigurer` plugins are read to be consistent with other hooks. Add a changelog entry.
> We decided to [reverse the order](https://github.com/ckan/ckan/blob/f2eb20ebf9de21016fd3d15ed028be1cce8d447c/ckan/config/environment.py#L118) in which `IConfigurer` plugins are read to be consistent with other hooks. Add a changelog entry.

Great, I can work on this one :) 

Resolving patch:
diff --git a/ckan/config/environment.py b/ckan/config/environment.py
--- a/ckan/config/environment.py
+++ b/ckan/config/environment.py
@@ -115,7 +115,7 @@ def update_config() -> None:
 
     webassets_init()
 
-    for plugin in p.PluginImplementations(p.IConfigurer):
+    for plugin in reversed(list(p.PluginImplementations(p.IConfigurer))):
         # must do update in place as this does not work:
         # config = plugin.update_config(config)
         plugin.update_config(config)


Test patch:
Repository: ckan/ckan

Issue:
Lock down cookie based authentication on API endpoints
Discussed on 2022-08-23 dev meeting:

1. Default behaviour: discard cookie based authentication in the API if it doesn't include the CSRF token
    - Is this equivalent to remove the `@csrf.exempt` decorator [here](https://github.com/ckan/ckan/blob/34401f000846aa59559c77e755e56656cf1355f9/ckan/views/api.py#L212)?
2. Add new configuration option: turn off entirely cookie based authentication on the API endpoint
    - Not sure how to implement this, but probably we need to fiddle at the Flask-login level. Maybe [this](https://flask-login.readthedocs.io/en/latest/#disabling-session-cookie-for-apis) is helpful?
3. (Optional): Allow cookie based authentication without valid CSRF tokens on development mode

@wardi @EricSoroos did I capture that correctly?


Discussion:
@TomeCirun you were kindly volunteered to work on this by @tino097 :)

Let's start with the first point. I think what we need to do is:

1. Remove the `@csrf.exempt` check from the API `action` endpoint. This will cause all client side JS API calls to fail, so
2. We need to update the JS calls to include the CSRF header, like you did for the `follow` module on #6863 . Remember to move the `addCsrfTokenHeader()` function to the [base module class](https://github.com/ckan/ckan/blob/master/ckan/public/base/javascript/module.js). Or maybe we even don't have to change any modules and just modify the [client class](https://github.com/ckan/ckan/blob/master/ckan/public/base/javascript/client.js) they use to add the header
3. Document the behaviour change (No cookie based authentication in the API without CSRF token) and the changes needed in JS if they are not using the ckan client

Does this make sense?

Resolving patch:
diff --git a/ckan/views/api.py b/ckan/views/api.py
--- a/ckan/views/api.py
+++ b/ckan/views/api.py
@@ -15,7 +15,6 @@
 
 import ckan.model as model
 from ckan.common import json, _, g, request, current_user
-from ckan.config.middleware.flask_app import csrf
 from ckan.lib.helpers import url_for
 from ckan.lib.base import render
 from ckan.lib.i18n import get_locales_from_config
@@ -208,8 +207,6 @@ def mixed(multi_dict: "MultiDict[str, Any]") -> dict[str, Any]:
 
 # View functions
 
-# exempt the CKAN API actions from csrf-validation.
-@csrf.exempt
 def action(logic_function: str, ver: int = API_DEFAULT_VERSION) -> Response:
     u'''Main endpoint for the action API (v3)
 


Test patch:
Repository: ckan/ckan

Issue:
Search indexing logic called twice after update or create dataset
**CKAN version**
2.10


**Describe the bug**

When updating or creating a dataset, we are indexing the dataset twice in a row, ie the [`index_package()`](https://github.com/ckan/ckan/blob/9f1b5cfaff8c135b589e2ea0275f1286c2e02711/ckan/lib/search/index.py#L108) function gets called twice during the same operation (and of course any `IPackageController.before_index()` hook gets called twice as well.

The root cause is the the obscure code run in the [`DomainObjectModificationExtension`](https://github.com/ckan/ckan/blob/9f1b5cfaff8c135b589e2ea0275f1286c2e02711/ckan/model/modification.py#L27), which considers the same Package object both new and changed at the same time, and fires two separate notification events.


Discussion:

Resolving patch:
diff --git a/ckan/model/modification.py b/ckan/model/modification.py
--- a/ckan/model/modification.py
+++ b/ckan/model/modification.py
@@ -47,8 +47,12 @@ def notify_observers(self, session: Any, method: Any):
                 for item in plugins.PluginImplementations(plugins.IResourceUrlChange):
                     item.notify(obj)
 
-        changed_pkgs = set(obj for obj in changed
-                           if isinstance(obj, model.Package))
+
+        changed_pkgs = set()
+        new_pkg_ids = [obj.id for obj in new if isinstance(obj, model.Package)]
+        for obj in changed:
+            if isinstance(obj, model.Package) and obj.id not in new_pkg_ids:
+                changed_pkgs.add(obj)
 
         for obj in new | changed | deleted:
             if not isinstance(obj, model.Package):


Test patch:
diff --git a/ckan/tests/lib/search/test_index.py b/ckan/tests/lib/search/test_index.py
--- a/ckan/tests/lib/search/test_index.py
+++ b/ckan/tests/lib/search/test_index.py
@@ -3,10 +3,12 @@
 import datetime
 import hashlib
 import json
+from unittest import mock
 import pytest
 import six
 from ckan.common import config
 import ckan.lib.search as search
+from ckan.tests import factories, helpers
 
 
 @pytest.mark.skipif(not search.is_available(), reason="Solr not reachable")
@@ -299,3 +301,22 @@ def test_indexed_package_stores_resource_type(self):
 
         # Resource types are indexed
         assert indexed_pkg["res_type"] == ["doc", "file"]
+
+
+@pytest.mark.usefixtures("clean_index")
+def test_index_only_called_once():
+
+    with mock.patch("ckan.lib.search.index.PackageSearchIndex.index_package") as m:
+
+        dataset = factories.Dataset()
+
+        assert m.call_count == 1
+
+        m.reset_mock()
+
+        dataset["notes"] = "hi"
+        helpers.call_action("package_update", **dataset)
+
+        assert m.call_count == 1
+
+        assert helpers.call_action("package_show", id=dataset["id"])["notes"] == "hi"
Repository: ckan/ckan

Issue:
resource_patch does not update file size
**CKAN version**
2.9.5, probably others

**Describe the bug**
Updating an uploaded resource file through API does not update the resource's `size` field

**Steps to reproduce**
Steps to reproduce the behavior:
1. Create a dataset and a file upload resource
1. Observe the resource's `size` field value
1. Change the file through the UI
1. Observe the resource's `size` field value changes as it should
1. Use the action API to upload a new file with `resource_patch` (or `resource_update`, I assume)
1. Observe the resource's `size` field value stays the same

**Expected behavior**
The file size should reflect the uploaded file regardless of the method used to upload it

**Additional details**
I assume this is because the resource already has a `size` field when `resource_update` calls  `package_update`, which assumes it is already set to the correct value. In `resource_create` the size is set, but `resource_update` just keeps the same value as before. Thus the new file size is not set.



Discussion:
Looks like we need to remove the `if 'size' not in resource` here?

https://github.com/ckan/ckan/blob/97d48e1d5ad0c52ba7352baa155bb320c74f0ae6/ckan/logic/action/update.py#L305

Do you want to give it a go and submit a PR if it works @bzar ?
Hi @amercader @bzar  , I would like to work on this issue.

@amercader seems to fix the issue with a quick test. Since @robin-NEC has not PR'd yet I'll do it.

Resolving patch:
diff --git a/ckan/logic/action/update.py b/ckan/logic/action/update.py
--- a/ckan/logic/action/update.py
+++ b/ckan/logic/action/update.py
@@ -302,7 +302,7 @@ def package_update(
             if hasattr(upload, 'mimetype'):
                 resource['mimetype'] = upload.mimetype
 
-        if 'size' not in resource and 'url_type' in resource:
+        if 'url_type' in resource:
             if hasattr(upload, 'filesize'):
                 resource['size'] = upload.filesize
 


Test patch:
Repository: ckan/ckan

Issue:
Helpers.current_url returns inconsistently quoted url, causing 500 errors.
**CKAN version**
Definitely 2.8, Probably 2.9, Not sure about master. Bug appears to have been added 5 years ago.  Definitely on python2, maybe on python3.

**Describe the bug**

https://site/%EF%AC%81?foo=bar&bz=%AC%81

`ckan.config.middleware.__init__.py:handle_i18n` urlencodes the querystring, but this function is explictly not for querystrings, as it encodes the special characters. (it encodes & and =, so It's for things that go into urls and querystrings, not the whole querystring).

This urlencoded querystring gets added back into the current url, at which point we have a current "url" which is potentially not a valid url for anything in our system.  As evidence for this, datastore/backend/postgres.py has added a patch to parse the returned url, unquote, and then reassemble.

Because of this, helpers.current_url returns a urldecoded version of the url, which is also not really something that we want to do, because that decodes the path component as well.

Now, once we're url decoding, we're taking it out of ascii and putting essentially arbitrary bytes through a unicode encode/decode process and potentially getting invalid UTF-8, or trying to put UTF-8 bytes into an ascii string, causing an error.


**Steps to reproduce**
Steps to reproduce the behavior:

Visit https://site/%EF%AC%81 and it returns a 500 error, not a 404.  The path is the fi ligature. 

**Expected behavior**
Return a 404, not a server error.



Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -655,7 +655,7 @@ def full_current_url() -> str:
 @core_helper
 def current_url() -> str:
     ''' Returns current url unquoted'''
-    return unquote(request.environ['CKAN_CURRENT_URL'])
+    return request.environ['CKAN_CURRENT_URL']
 
 
 @core_helper
diff --git a/ckan/views/__init__.py b/ckan/views/__init__.py
--- a/ckan/views/__init__.py
+++ b/ckan/views/__init__.py
@@ -190,8 +190,6 @@ def set_ckan_current_url(environ: Any) -> None:
 
     qs = environ.get(u'QUERY_STRING')
     if qs:
-        # sort out weird encodings
-        qs = quote(qs, u'')
         environ[u'CKAN_CURRENT_URL'] = u'%s?%s' % (path_info, qs)
     else:
         environ[u'CKAN_CURRENT_URL'] = path_info
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -21,7 +21,7 @@
 
 import six
 from urllib.parse import (
-    urlencode, unquote, urlunparse, parse_qsl, urlparse
+    urlencode, urlunparse, parse_qsl, urlparse
 )
 from io import StringIO
 
@@ -764,7 +764,7 @@ def _insert_links(data_dict: dict[str, Any], limit: int, offset: int):
 
     # change the offset in the url
     parsed = list(urlparse(urlstring))
-    query = unquote(parsed[4])
+    query = parsed[4]
 
     arguments = dict(parse_qsl(query))
     arguments_start = dict(arguments)


Test patch:
Repository: ckan/ckan

Issue:
Hyperlink with the UK Open Government License does not work
**CKAN version**
2.9.5

**Describe the bug**
The link in the `license.py` under class `LicenseOpenGovernment`, stating (http://reference.data.gov.uk/id/open-government-licence) is now a non-existing page.

It is rendered on the dataset using the license, which is going to an invalid link.

Found the recent one should be provided by [The National Archives](https://www.nationalarchives.gov.uk/doc/open-government-licence)


**Steps to reproduce**
1. Have a dataset with uk-ogl license
2. Go to the package view page and the link for the license is there

**Expected behavior**
Expected reaching to a valid link, for example the [The National Archives](https://www.nationalarchives.gov.uk/doc/open-government-licence) one

**Additional details**
It is controlled under class `LicenseOpenGovernment` in `ckan > ckan > model > license.py`



Discussion:

Resolving patch:
diff --git a/ckan/model/license.py b/ckan/model/license.py
--- a/ckan/model/license.py
+++ b/ckan/model/license.py
@@ -303,7 +303,7 @@ class LicenseOpenGovernment(DefaultLicense):
     id = "uk-ogl"
     od_conformance = 'approved'
     # CS: bad_spelling ignore
-    url = "http://reference.data.gov.uk/id/open-government-licence"
+    url = "https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/"
 
     @property
     def title(self):


Test patch:
Repository: ckan/ckan

Issue:
Theme regression in 2.9.6
**CKAN version**
2.9.6

**Describe the bug**
On updating to ckan 2.9.6 from 2.9.5, an extension that used the app global main_css failed with a jinja template error:
```
ckan29                        |     {% asset main_css[6:-4] %}
ckan29                        | jinja2.exceptions.UndefinedError: 'ckan.config.middleware.flask_app.CKAN_AppCtxGlobals object' has no attribute 'main_css'
```

Ref: https://github.com/ckan/ckan/commit/f9084f909538a963b1121d5cfb1e3a1485324add

**Steps to reproduce**
Steps to reproduce the behavior:
Have an extension that refers to the old `main_css` asset or `h.get_rtl_css`

**Expected behavior**
Not an error -- point releases should not remove portions of the publicly visible API.




Discussion:

Resolving patch:
diff --git a/ckan/lib/app_globals.py b/ckan/lib/app_globals.py
--- a/ckan/lib/app_globals.py
+++ b/ckan/lib/app_globals.py
@@ -91,6 +91,14 @@ def set_theme(asset):
     app_globals.theme = asset
 
 
+def set_main_css(css_file):
+    ''' Sets the main_css.  The css_file must be of the form file.css '''
+    assert css_file.endswith('.css')
+    new_css = css_file
+    # FIXME we should check the css file exists
+    app_globals.main_css = str(new_css)
+
+
 def set_app_global(key, value):
     '''
     Set a new key on the app_globals (g) object
@@ -181,6 +189,9 @@ def get_config_value(key, default=''):
     # custom styling
     theme = get_config_value('ckan.theme') or 'css/main'
     set_theme(theme)
+    # legacy -- preserve for extensinos
+    main_css = get_config_value('ckan.main_css', '/base/css/main.css')
+    set_main_css(main_css)
 
     if app_globals.site_logo:
         app_globals.header_class = 'header-image'
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -691,6 +691,11 @@ def get_rtl_theme():
     return config.get('ckan.i18n.rtl_theme', 'css/main-rtl')
 
 
+@core_helper
+def get_rtl_css():
+    return config.get('ckan.i18n.rtl_css', '/base/css/main-rtl.css')
+
+
 class Message(object):
     '''A message returned by ``Flash.pop_messages()``.
 


Test patch: