internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
Is ckan.datasets_per_page not being used?
**CKAN version**
CKAN 2.9.5

**Describe the bug**
I set the _ckan.datasets_per_page_ configuration options to 10 and this is working fine on routes:
- /dataset/id
- /group
- /organization

 But is not working in the routes*
- /group/id
- /organization/id

where it always shows 20 results per page. Reading through the source code I noticed that the _read_ function called in *those routes receives a _limit_ parameter with a default value of 20, but it is not updated in the function body with something like

```python
the_limit = config.get('ckan.datasets_per_page', limit)
```
I added this change in my local installation and it worked. Is this by design or is it a bug? I just want to be sure that this is a bug before opening a PR.

https://github.com/ckan/ckan/blob/dc49cece33e447858d0008bc71b9a4ab724602ee/ckan/views/group.py#L408-L411


Thanks. 


Discussion:
@ddazal That makes sense, want to submit a PR?

Resolving patch:
diff --git a/ckan/views/group.py b/ckan/views/group.py
--- a/ckan/views/group.py
+++ b/ckan/views/group.py
@@ -425,6 +425,8 @@ def read(group_type: str,
 
     extra_vars["q"] = q
 
+    limit = config.get(u'ckan.datasets_per_page', limit)
+
     try:
         # Do not query for the group datasets when dictizing, as they will
         # be ignored and get requested on the controller anyway


Test patch:
Repository: ckan/ckan

Issue:
Add an index in table resource_view to improve DB performance
**CKAN version**
2.9.6, master

**Describe the bug**
Table `resource_view` needs a index on column `resource_id`.

**Steps to reproduce**
In a big CKAN database (500k packages) run the query
```
begin;
explain (analyze,buffers,timing)
delete from resource where package_id = [###];                                  
rollback;
...
Trigger for constraint resource_view_resource_id_fkey: time=68.124 calls=6
```

**Expected behavior**
```
CREATE INDEX idx_view_resource_id
ON resource_view(resource_id);
```
After the index added, the time dropped from **68.124** to **0.420**, that is 170 times faster.

**Additional details**
This will greatly reduce time spent when clearing a big harvest source.


Discussion:

Resolving patch:
diff --git a/ckan/migration/versions/104_9f33a0280c51_resource_view_resource_id_index.py b/ckan/migration/versions/104_9f33a0280c51_resource_view_resource_id_index.py
new file mode 100644
--- /dev/null
+++ b/ckan/migration/versions/104_9f33a0280c51_resource_view_resource_id_index.py
@@ -0,0 +1,27 @@
+# encoding: utf-8
+
+"""resource_view resource_id index
+
+Revision ID: 9f33a0280c51
+Revises: 353aaf2701f0
+Create Date: 2022-10-11 02:31:35.941858
+
+"""
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision = '9f33a0280c51'
+down_revision = '353aaf2701f0'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.create_index(
+        u'idx_view_resource_id', u'resource_view', [u'resource_id']
+    )
+
+
+def downgrade():
+    op.drop_index(u'idx_view_resource_id')


Test patch:
Repository: ckan/ckan

Issue:
allow plugins to implent old and new method names
**CKAN version**
master, 2.10

**Describe the bug**
In the past it has been possible for plugins to implement both the old and new names of methods, but since #7148 the new method gets overwritten by the old method.

**Steps to reproduce**
Use e.g. scheming_nerf_index plugin which implements both `before_index` and `before_dataset_index` for compatibility

**Expected behavior**
Not an infinite loop :upside_down_face: 

**Additional details**




Discussion:

Resolving patch:
diff --git a/ckan/plugins/core.py b/ckan/plugins/core.py
--- a/ckan/plugins/core.py
+++ b/ckan/plugins/core.py
@@ -157,7 +157,7 @@ def __init__(self, *args: Any, **kwargs: Any):
                 ["after_search", "after_dataset_search"],
                 ["before_index", "before_dataset_index"],
                     ["before_view", "before_dataset_view"]]:
-                if hasattr(self, old_name):
+                if hasattr(self, old_name) and not hasattr(self, new_name):
                     msg = (
                         f"The method 'IPackageController.{old_name}' is "
                         + f"deprecated. Please use '{new_name}' instead!"
@@ -175,7 +175,7 @@ def __init__(self, *args: Any, **kwargs: Any):
                 ["before_delete", "before_resource_delete"],
                 ["after_delete", "after_resource_delete"],
                     ["before_show", "before_resource_show"]]:
-                if hasattr(self, old_name):
+                if hasattr(self, old_name) and not hasattr(self, new_name):
                     msg = (
                         f"The method 'IResourceController.{old_name}' is "
                         + f"deprecated. Please use '{new_name}' instead!"


Test patch:
Repository: ckan/ckan

Issue:
UI broken in Resource DataStore tab
**CKAN version**
2.10/master

**Describe the bug**
Missing icons and misaligned

![Screenshot 2022-11-28 at 11-38-11 dadada - 25_best_tacos_austin csv - CKAN](https://user-images.githubusercontent.com/200230/204535911-8e94acc1-7eed-4dc9-b7a1-d06891874274.png)







Discussion:
Probably a similar fix as https://github.com/ckan/ckan/pull/7169
A little bit more of context: The problem here is that the timeline depends on a css class `activity` that is no longer available in CKAN core since it has been moved to the `activity` extension.

The solution is to rebuild the html/css of the timeline to look similar to the activity stream. #7169 contains the `css` class removed and how to fix the icon display using Font Awesome stack (https://fontawesome.com/docs/web/style/stack).

This change should be done in datapusher extension (and if time, also xloader one.)

To summarise:
 - `datapusher` extension should contain all the CSS files required to display that log timeline properly (either by mimicking activity's CSS or building a new one)
 - `xloader` will have the same issue, but that should be fixed in its repository.
 
 ### Notes on the extensions
 
* Both `xloader` and `datapusher` define a new view called `resource_data` that is going to render `resource_data.html`. This template {% extends "package/resource_edit_base.html" %}. (in both extensions) Neither extension overrides any other template. They just have the same behaviour (A view rendering a template with the same name). So `xloader` **is not** overriding `datapusher` template it just render a similar template in the same position.
* `xloader` is a replacement for `datapusher` so only one should be available at the time. If the two are active, then two Datastore tabs are gonna be displayed. This is a side effect of both extension having a similar behaviour (defining a view and rendering a template in the same location).

Resolving patch:
diff --git a/ckanext/datapusher/plugin.py b/ckanext/datapusher/plugin.py
--- a/ckanext/datapusher/plugin.py
+++ b/ckanext/datapusher/plugin.py
@@ -37,6 +37,8 @@ class DatapusherPlugin(p.SingletonPlugin):
     def update_config(self, config: CKANConfig):
         templates_base = config.get_value(u'ckan.base_templates_folder')
         p.toolkit.add_template_directory(config, templates_base)
+        p.toolkit.add_public_directory(config, 'public')
+        p.toolkit.add_resource('assets', 'ckanext-datapusher')
 
     def configure(self, config: CKANConfig):
         self.config = config


Test patch:
Repository: ckan/ckan

Issue:
White on Yellow?? Color Contrast
**CKAN version**
https://demo.ckan.org/en/user/edit/mgifford

**Describe the bug**
Title: WCAG 1.4.3: Ensures the contrast between foreground and background colors meets WCAG 2 AA contrast ratio thresholds (.btn-warning)
Tags: Accessibility, WCAG 1.4.3, color-contrast

Issue: Ensures the contrast between foreground and background colors meets WCAG 2 AA contrast ratio thresholds (color-contrast - https://accessibilityinsights.io/info-examples/web/color-contrast)

Target application: Manage - mgifford - Users - CKAN Demo - https://demo.ckan.org/en/user/edit/mgifford

Element path: .btn-warning

Snippet: <a class="btn btn-warning" href="/en/user/generate_key/b8037a86-a216-4c9b-8211-e197fa09143a" data-module="confirm-action" data-module-content="Are you sure you want to regenerate the API key?">Regenerate API Key</a>

How to fix: 
Fix any of the following:
  Element has insufficient color contrast of 1.94 (foreground color: #ffffff, background color: #f0ad4e, font size: 10.5pt (14px), font weight: bold). Expected contrast ratio of 4.5:1

Environment: Microsoft Edge version 107.0.1418.35

====

This accessibility issue was found using Accessibility Insights for Web 2.35.0 (axe-core 4.4.1), a tool that helps find and fix accessibility issues. Get more information & download this tool at http://aka.ms/AccessibilityInsights.


Discussion:

Resolving patch:
diff --git a/ckan/cli/sass.py b/ckan/cli/sass.py
--- a/ckan/cli/sass.py
+++ b/ckan/cli/sass.py
@@ -11,21 +11,32 @@
 
 
 @click.command(
-    name=u'sass',
-    short_help=u'Compile all root sass documents into their CSS counterparts')
-def sass():
-    command = (u'npm', u'run', u'build')
-
-    public = config.get_value(u'ckan.base_public_folder')
-
-    root = os.path.join(os.path.dirname(__file__), u'..', public, u'base')
+    name='sass',
+    short_help='Compile all root sass documents into their CSS counterparts')
+@click.option(
+    '-d',
+    '--debug',
+    is_flag=True,
+    help="Compile css with sourcemaps.")
+def sass(debug: bool):
+    command = ('npm', 'run', 'build')
+
+    public = config.get_value('ckan.base_public_folder')
+
+    root = os.path.join(os.path.dirname(__file__), '..', public, 'base')
     root = os.path.abspath(root)
-    _compile_sass(root, command, u'main')
-
-
-def _compile_sass(root: str, command: tuple[str, ...], color: str):
-    click.echo(u'compile {}.css'.format(color))
-    command = command + (u'--', u'--' + color)
+    _compile_sass(root, command, 'main', debug)
+
+
+def _compile_sass(
+        root: str,
+        command: tuple[str, ...],
+        color: str,
+        debug: bool):
+    click.echo('compile {}.css'.format(color))
+    command = command + ('--', '--' + color)
+    if debug:
+        command = command + ('--debug',)
 
     process = subprocess.Popen(
         command,


Test patch:
Repository: ckan/ckan

Issue:
No default for WTF_CSRF_SECRET_KEY config option
## CKAN version
2.10 


## Describe the bug
The `WTF_CSRF_SECRET_KEY` config option, used to generate the CSRF tokens is defined as such in the declaration:

https://github.com/ckan/ckan/blob/9e60350f27c9266b5afbd741484265c5e06a2d38/ckan/config/config_declaration.yaml#L645-L647

This means that when a new ini file is generated, a secret value is added to the ini file by default, but when that config option is not defined in an ini file (ie you are using an ini file generated with a previous version of CKAN when upgrading your instance) the default value is `None`.

This causes the app to fail on all requests with:

```
  File "/home/adria/dev/pyenvs/ckan-py3/src/ckan/ckan/templates/base.html", line 27, in block 'meta'
    <meta name="{{ g.csrf_field_name }}" content="{{ csrf_token() }}" />
  File "/home/adria/dev/pyenvs/ckan-py3/lib/python3.8/site-packages/flask_wtf/csrf.py", line 36, in generate_csrf
    secret_key = _get_config(
  File "/home/adria/dev/pyenvs/ckan-py3/lib/python3.8/site-packages/flask_wtf/csrf.py", line 136, in _get_config
    raise RuntimeError(message)
RuntimeError: A secret key is required to use CSRF.
```
That's because wtforms falls back to Flask's `SECRET_KEY` only if `WTF_CSRF_SECRET_KEY` is not present in the config, but we add it to the config, just empty.

This setting should have either be checked at startup to see if it's set or fallback to a default like we do with the rest of the secret keys (I prefer the latter)



Discussion:
We need to come up with a consistent approach to handle secrets and their defaults, I'll raise that in a separate issue

Resolving patch:
diff --git a/ckan/config/middleware/flask_app.py b/ckan/config/middleware/flask_app.py
--- a/ckan/config/middleware/flask_app.py
+++ b/ckan/config/middleware/flask_app.py
@@ -262,7 +262,10 @@ def make_flask_stack(conf: Union[Config, CKANConfig]) -> CKANApp:
     _register_error_handler(app)
 
     # CSRF
-    app.config['WTF_CSRF_FIELD_NAME'] = "_csrf_token"
+    wtf_key = "WTF_CSRF_SECRET_KEY"
+    if not app.config.get(wtf_key):
+        config[wtf_key] = app.config[wtf_key] = app.config["SECRET_KEY"]
+    app.config["WTF_CSRF_FIELD_NAME"] = "_csrf_token"
     csrf.init_app(app)
 
     # Set up each IBlueprint extension as a Flask Blueprint


Test patch:
diff --git a/ckan/tests/config/test_middleware.py b/ckan/tests/config/test_middleware.py
--- a/ckan/tests/config/test_middleware.py
+++ b/ckan/tests/config/test_middleware.py
@@ -100,3 +100,13 @@ def test_flask_config_values_are_parsed(app):
     assert (
         app.flask_app.config["REMEMBER_COOKIE_DURATION"] == 12345
     )
+
+
+@pytest.mark.ckan_config("WTF_CSRF_SECRET_KEY", None)
+def test_no_wtf_secret_falls_back_to_secret_key(app):
+    assert (
+        app.flask_app.config["WTF_CSRF_SECRET_KEY"] == config.get("beaker.session.secret")
+    )
+    assert (
+        config["WTF_CSRF_SECRET_KEY"] == config.get("beaker.session.secret")
+    )
Repository: ckan/ckan

Issue:
datatablesview: show columns feature bug
## CKAN version
master, 2.10 (earlier?)

## Describe the bug
If any columns are unselected when creating a view, the view will not appear.

### Steps to reproduce
When creating or editing a datatablesview (not viewing an existing one) un-check some columns and save the view.

### Expected behavior
Those columns should be excluded but view should still work.

### Additional details





Discussion:

Resolving patch:
diff --git a/ckanext/datastore/helpers.py b/ckanext/datastore/helpers.py
--- a/ckanext/datastore/helpers.py
+++ b/ckanext/datastore/helpers.py
@@ -206,9 +206,13 @@ def _get_subquery_from_crosstab_call(ct: str):
     return ct.replace("''", "'")
 
 
-def datastore_dictionary(resource_id: str):
+def datastore_dictionary(
+        resource_id: str, include_columns: Optional[list[str]] = None):
     """
-    Return the data dictionary info for a resource
+    Return the data dictionary info for a resource, optionally filtering
+    columns returned.
+
+    include_columns is a list of column ids to include in the output
     """
     try:
         return [
@@ -217,6 +221,8 @@ def datastore_dictionary(resource_id: str):
                     u'resource_id': resource_id,
                     u'limit': 0,
                     u'include_total': False})['fields']
-            if not f['id'].startswith(u'_')]
+            if not f['id'].startswith(u'_') and (
+                include_columns is None or f['id'] in include_columns)
+            ]
     except (tk.ObjectNotFound, tk.NotAuthorized):
         return []


Test patch:
Repository: ckan/ckan

Issue:
2.10.0 - Link to CKAN-API in footer points to 404
## CKAN version
2.10.0

## Describe the bug
CKAN-API link in the footer points to 404 site: http://docs.ckan.org/en/2.1/api/
![ckan_api_404](https://user-images.githubusercontent.com/414984/220368786-68cac0db-552c-4265-b65a-9f4e73cfd8c3.png)

### Steps to reproduce
See/Click the CKAN-API link in the footer.

### Expected behavior
Landing on a working site with details about the CKAN-API, probably: https://docs.ckan.org/en/2.10/api/

### Additional details
*




Discussion:

Resolving patch:
diff --git a/ckan/lib/app_globals.py b/ckan/lib/app_globals.py
--- a/ckan/lib/app_globals.py
+++ b/ckan/lib/app_globals.py
@@ -7,6 +7,7 @@
 import re
 from threading import Lock
 from typing import Any, Union
+from packaging.version import parse as parse_version
 
 import ckan
 import ckan.model as model
@@ -14,6 +15,7 @@
 from ckan.common import asbool, config, aslist
 
 
+
 log = logging.getLogger(__name__)
 
 
@@ -219,11 +221,12 @@ def _check_uptodate(self):
     def _init(self):
 
         self.ckan_version = ckan.__version__
-        self.ckan_base_version = re.sub(r'[^0-9\.]', '', self.ckan_version)
-        if self.ckan_base_version == self.ckan_version:
-            self.ckan_doc_version = self.ckan_version[:3]
+        version = parse_version(self.ckan_version)
+        self.ckan_base_version = version.base_version
+        if not version.is_prerelease:
+            self.ckan_doc_version = f"{version.major}.{version.minor}"
         else:
-            self.ckan_doc_version = 'latest'
+            self.ckan_doc_version = "latest"
         # process the config details to set globals
         for key in app_globals_from_config_details.keys():
             new_key, value = process_app_global(key, config.get(key) or '')


Test patch:
Repository: ckan/ckan

Issue:
Default to client-based sessions for new sites
The Flask-Login refactor introduced changes regarding logged in users and the session. To re-cap:
Flask-login stores the logged-in user identifier in the Flask session (as opposed to the `auth_tkt` in CKAN<2.10). CKAN uses [Beaker](https://beaker.readthedocs.io/en/latest/sessions.html) to manage the session, and the default session backend stores this session information as files on the server (on `/tmp`). This means that **if the session data is deleted in the server, all users will be logged out of the site**.
This can happen for instance:
* if the CKAN container is redeployed in a Docker / cloud setup and the session directory is not persisted
* if the sessions are periodically cleaned by an external script

| Action | CKAN < 2.10 | CKAN >= 2.10 |
| ------ | ----------- | ------------ |
| Clear cookies | User logged out | User logged out (If `remember_me` cookie is deleted) |
| Clear server sessions | User still logged in | User logged out |

The way to keep the old behaviour with the Beaker backend is to store the session data in the [cookie itself](https://beaker.readthedocs.io/en/latest/sessions.html#cookie-based) (but this stores *all* session data, not just the user identifier):

```
# ckan.ini
beaker.session.type = cookie
beaker.session.validate_key = tT0ka0aOAWEPbvSOSog7c4ZFu

# These should probably be defaults anyway
beaker.session.httponly = True
beaker.session.secure = True
beaker.session.samesite = Lax # or Strict
```

Going through the Flask-login discussion, it looks like we decided to default to storing the session in the cookie on new instances ( https://github.com/ckan/ckan/pull/6560#issuecomment-1161693905). We didn't end up implementing this in CKAN 2.10, should we aim for it in CKAN 2.11?

If we go ahead this will probably just involve configuration and documentation changes.



Discussion:
Decisions from the dev meeting:

* Aim to implement this change in 2.10.1, not 2.11. It should only involve configuration and documentation changes and only affect new sites.
* CHANGELOG should clearly document the new default configuration in case existing sites want to replicate it. Things to make sure to mention:
  * Potential issues with the cookie size if users store large amounts of data in the session
  * `beaker.session.samesite` needs to be `Lax` if using some third-party Auth otherwise the cookie will get dropped (@Zharktas please expand/correct)
  * Users can always set up server side sessions with redis or DB if necessary

Resolving patch:
diff --git a/ckan/config/declaration/option.py b/ckan/config/declaration/option.py
--- a/ckan/config/declaration/option.py
+++ b/ckan/config/declaration/option.py
@@ -366,6 +366,7 @@ def _validators_from_string(s: str) -> list[Validator]:
 
     out = []
     parts = s.split()
+
     for p in parts:
         if '(' in p and p[-1] == ')':
             name, args = p.split('(', 1)


Test patch:
Repository: ckan/ckan

Issue:
Mailer doesn't quote names in To/From headers
## CKAN version
2.9.8, though it doesn't appear to have changed on 2.10
## Describe the bug
The username and site title are included in the To/From headers respectively, but they aren't quoted. When one contains, for example, commas, it's displayed as several addresses.
(When site_title is `Dept of A, B, and C`, the emails appear to be from `Dept@{host}, of@{host}, A@{host}, B@{host}, and C,`)
### Steps to reproduce
Steps to reproduce the behavior:
Set a site_title to a string containing commas and trigger an email somehow. I've been using `ckanext-harvester` notifications.
### Expected behavior
That the name would be quoted and therefore show as `Dept of A, B, and C`.

### Additional details
#4343 appeared to address the same / a similar issue




Discussion:

Resolving patch:
diff --git a/ckan/lib/mailer.py b/ckan/lib/mailer.py
--- a/ckan/lib/mailer.py
+++ b/ckan/lib/mailer.py
@@ -64,8 +64,8 @@ def _mail_recipient(
         else:
             msg.add_header(k, v)
     msg['Subject'] = subject
-    msg['From'] = _("%s <%s>") % (sender_name, mail_from)
-    msg['To'] = u"%s <%s>" % (recipient_name, recipient_email)
+    msg['From'] = utils.formataddr((sender_name, mail_from))
+    msg['To'] = utils.formataddr((recipient_name, recipient_email))
     msg['Date'] = utils.formatdate(time())
     if not config.get('ckan.hide_version'):
         msg['X-Mailer'] = "CKAN %s" % ckan.__version__


Test patch:
diff --git a/ckan/tests/lib/test_mailer.py b/ckan/tests/lib/test_mailer.py
--- a/ckan/tests/lib/test_mailer.py
+++ b/ckan/tests/lib/test_mailer.py
@@ -6,6 +6,7 @@
 from email.header import decode_header
 from email.mime.text import MIMEText
 from email.parser import Parser
+import email.utils
 
 import ckan.lib.helpers as h
 import ckan.lib.mailer as mailer
@@ -199,10 +200,10 @@ def test_from_field_format(self, mail_server):
         msgs = mail_server.get_smtp_messages()
         msg = msgs[0]
 
-        expected_from_header = "{0} <{1}>".format(
+        expected_from_header = email.utils.formataddr((
             config.get("ckan.site_title"),
             config.get("smtp.mail_from")
-        )
+        ))
 
         assert expected_from_header in msg[3]
Repository: ckan/ckan

Issue:
Activity stream plugin on CKAN 2.10 does not redirect unauthenticated dashboard to login page
## CKAN version

2.10

## Describe the bug

The dashboard views are meant to redirect to the login page if the user is not authenticated. However, the activity stream plugin does not. Instead, it encounters a "User not found" error.

### Steps to reproduce

- Start a CKAN 2.10 instance with the 'activity' plugin enabled.
- Go to /dashboard/datasets without logging in. You should see the login page.
- Go to /dashboard without logging in. You should see the login page but will actually see an error.

### Expected behavior

The /dashboard URL should redirect to the login page if not logged in.

### Additional details

The error message appears to originate from `ckan.views.user._extra_template_variables`; the activity plugin is attempting to preload user information.




Discussion:

Resolving patch:
diff --git a/ckanext/activity/views.py b/ckanext/activity/views.py
--- a/ckanext/activity/views.py
+++ b/ckanext/activity/views.py
@@ -810,6 +810,10 @@ def user_activity(id: str) -> str:
 
 @bp.route("/dashboard/", strict_slashes=False)
 def dashboard() -> str:
+    if tk.current_user.is_anonymous:
+        tk.h.flash_error(tk._(u'Not authorized to see this page'))
+        return tk.h.redirect_to(u'user.login')
+
     context = cast(
         Context,
         {


Test patch: