Repository: ckan/ckan

Issue:
Email notifications documentation not working on CKAN 2.9.x
**CKAN version: 2.9.x**

**Describe the bug**
The documentation regarding email notifications https://docs.ckan.org/en/2.9/maintaining/email-notifications.html#email-notifications is not working on 2.9.x due to nothing replacing `paster post`

**Steps to reproduce**
Try running `ckan post`, command post not available.

**Expected behavior**
An authenticated API request to the send_email_notifications action is made.


Discussion:
@amercader also struggling with the same issue. I think the 'ckan post' command has been replaced with the 'ckan notify' but unable to find any info on usage. 
Could you assist with this issue?

Cheers,
Ammar
@mbocevski have you been able to sort this issue yet? I am also trying to work this one out. I think the 'ckan post' command has been replaced with the 'ckan notify' but unable to find any info on usage.
Notify is a different thing. Post was internal functionality of paster, which needs to implemented in ckan cli as paster was removed.
> Notify is a different thing. Post was internal functionality of paster, which needs to implemented in ckan cli as paster was removed.

@Zharktas thanks for the quick response. Any work around for sending email notifications until this is implemented by the ckan team?
There is no workaround, someone needs to implement it.
>have you been able to sort this issue yet?

>There is no workaround

@ammarfmg yes there is a workaround. What `ckan post` does is basically sends a sysadmin authenticated request to the `/api/action/send_email_notifications` endpoint. So you can implement it simply by registering a sysadmin account, generating an API token and then using that token you can call something like:
 `curl -s -H "Authorization: SYSADMIN_API_TOKEN" -d {} http://<ckan_url>/api/action/send_email_notifications`

You can see how we've implemented it on our ckan helm package here: https://github.com/keitaroinc/ckan-helm/blob/master/templates/cronjob.yaml#L19-L26

> > have you been able to sort this issue yet?
> 
> > There is no workaround
> 
> @ammarfmg yes there is a workaround. What `ckan post` does is basically sends a sysadmin authenticated request to the `/api/action/send_email_notifications` endpoint. So you can implement it simply by registering a sysadmin account, generating an API token and then using that token you can call something like:
> `curl -s -H "Authorization: SYSADMIN_API_TOKEN" -d {} http://<ckan_url>/api/action/send_email_notifications`
> 
> You can see how we've implemented it temporarily on our ckan helm package here: https://github.com/keitaroinc/ckan-helm/blob/master/templates/cronjob.yaml#L19-L26

@mbocevski thanks a lot for this. I will give it a try. 
@mbocevski after generating an API key, can I add the curl command to my crontab file with an @hourly keyword?

@hourly curl -s -H "Authorization: SYSADMIN_API_TOKEN" -d {} http://<ckan_url>/api/action/send_email_notifications
>can I add the curl command to my crontab file

Of course.
Thanks again. It is running but giving this message:

"Bad request - JSON Error: Invalid request. Please use POST method for your request"

I will try to solve it. 
>"Bad request - JSON Error: Invalid request. Please use POST method for your request"

Did you include the `-d {}` that means to send empty data with the request which will automatically send a POST request. You can always add `-X POST` if you want to explicitly set the request method.
Try running the curl command from the shell to see that all works fine. You don't need to use `sudo`. It shouldn't matter but you have double forward-slashes `//api`. If it works fine from the shell, and doesn't work from cron then check whether you need to quote the command or something depends on your systems cron.
Ok sure. I am getting the following output in shell:

{"help": "http://x.x.x.x/api/3/action/help_show?name=send_email_notifications", "success": true, "result": null}

Is this the expected behavior?
>{"help": "http://x.x.x.x/api/3/action/help_show?name=send_email_notifications", "success": true, "result": null}
>Is this the expected behavior?

Yes! Great.
> > {"help": "http://x.x.x.x/api/3/action/help_show?name=send_email_notifications", "success": true, "result": null}
> > Is this the expected behavior?
> 
> Yes! Great.

Thank you and have a good day. 
> There is no workaround, someone needs to implement it.

@Zharktas What do you think to implement it in the [notify CLI](https://github.com/ckan/ckan/blob/master/ckan/cli/notify.py) itself with the new click command as `send_email_notifications` which will trigger the notifications

Something like this

```python
@notify.command(
    name=u'send_email_notifications',
    short_help=u'Send out Email notifications.'
)
def send_email_notifications():
    if not converters.asbool(
            config.get('ckan.activity_streams_email_notifications')):
        raise ValidationError('ckan.activity_streams_email_notifications'
                              ' is not enabled in config')
    email_notifications.get_and_send_notifications_for_all_users()

```


WDYT about the above approach?

> WDYT about the above approach?

I mostly like it, but if it's already getting filed under 'notify', maybe it can have a shorter name?

NB Putting it inside a Click command would move the overhead from the CKAN server to the location where the command is being run; often they are the same, but they don't have to be.

Resolving patch:
diff --git a/ckan/cli/notify.py b/ckan/cli/notify.py
--- a/ckan/cli/notify.py
+++ b/ckan/cli/notify.py
@@ -1,23 +1,44 @@
 # encoding: utf-8
 
+from logging import getLogger
 import click
 from ckan.model import Session, Package, DomainObjectOperation
 from ckan.model.modification import DomainObjectModificationExtension
+from ckan.logic import NotAuthorized, ValidationError
 
 
-@click.group(
-    name=u'notify',
-    short_help=u'Send out modification notifications.'
-)
+log = getLogger(__name__)
+
+
+@click.group(name="notify", short_help="Send out modification notifications.")
 def notify():
     pass
 
 
-@notify.command(
-    name=u'replay',
-    short_help=u'Send out modification signals.'
-)
+@notify.command(name="replay", short_help="Send out modification signals.")
 def replay():
     dome = DomainObjectModificationExtension()
     for package in Session.query(Package):
         dome.notify(package, DomainObjectOperation.changed)
+
+
+@notify.command(name="send_emails", short_help="Send out Email notifications.")
+def send_emails():
+    """ Sends an email to users notifying about new activities.
+
+    As currently implemented, it will only send notifications from dashboard
+    activity list if users have `activity_streams_email_notifications` set
+    in their profile. It will send emails with updates depending
+    on the `ckan.email_notifications_since` config. (default: 2 days.)
+    """
+    import ckan.logic as logic
+    import ckan.lib.mailer as mailer
+    from ckan.types import Context
+    from typing import cast
+
+    site_user = logic.get_action("get_site_user")({"ignore_auth": True}, {})
+    context = cast(Context, {"user": site_user["name"]})
+    try:
+        logic.get_action("send_email_notifications")(context, {})
+    except (NotAuthorized, ValidationError, mailer.MailerException) as e:
+        log.error(e)
diff --git a/ckan/lib/email_notifications.py b/ckan/lib/email_notifications.py
--- a/ckan/lib/email_notifications.py
+++ b/ckan/lib/email_notifications.py
@@ -11,12 +11,13 @@
 import datetime
 import re
 from typing import Any, cast
+from jinja2 import Environment
 
 import ckan.model as model
 import ckan.logic as logic
-import ckan.lib.base as base
+import ckan.lib.jinja_extensions as jinja_extensions
 
-from ckan.common import ungettext, config
+from ckan.common import ungettext, ugettext, config
 from ckan.types import Context
 
 
@@ -77,6 +78,18 @@ def string_to_timedelta(s: str) -> datetime.timedelta:
     return delta
 
 
+def render_activity_email(activities: list[dict[str, Any]]) -> str:
+    globals = {'site_title': config.get_value('ckan.site_title')}
+    template_name = 'activity_streams/activity_stream_email_notifications.text'
+
+    env = Environment(**jinja_extensions.get_jinja_env_options())
+    # Install the given gettext, ngettext callables into the environment
+    env.install_gettext_callables(ugettext, ungettext) # type: ignore
+
+    template = env.get_template(template_name, globals=globals)
+    return template.render({'activities': activities})
+
+
 def _notifications_for_activities(
         activities: list[dict[str, Any]],
         user_dict: dict[str, Any]) -> list[dict[str, str]]:
@@ -105,15 +118,15 @@ def _notifications_for_activities(
     # say something about the contents of the activities, or single out
     # certain types of activity to be sent in their own individual emails,
     # etc.
+
     subject = ungettext(
         "{n} new activity from {site_title}",
         "{n} new activities from {site_title}",
         len(activities)).format(
                 site_title=config.get_value('ckan.site_title'),
                 n=len(activities))
-    body = base.render(
-            'activity_streams/activity_stream_email_notifications.text',
-            extra_vars={'activities': activities})
+
+    body = render_activity_email(activities)
     notifications = [{
         'subject': subject,
         'body': body
@@ -214,7 +227,6 @@ def get_and_send_notifications_for_user(user: dict[str, Any]) -> None:
     email_last_sent = model.Dashboard.get(user['id']).email_last_sent
     activity_stream_last_viewed = (
             model.Dashboard.get(user['id']).activity_stream_last_viewed)
-
     since = max(email_notifications_since, email_last_sent,
             activity_stream_last_viewed)
 
diff --git a/ckan/logic/action/update.py b/ckan/logic/action/update.py
--- a/ckan/logic/action/update.py
+++ b/ckan/logic/action/update.py
@@ -1140,8 +1140,8 @@ def dashboard_mark_activities_old(
 def send_email_notifications(context: Context, data_dict: DataDict) -> ActionResult.SendEmailNotifications:
     '''Send any pending activity stream notification emails to users.
 
-    You must provide a sysadmin's API key in the Authorization header of the
-    request, or call this action from the command-line via a `paster post ...`
+    You must provide a sysadmin's API key/token in the Authorization header of the
+    request, or call this action from the command-line via a `ckan notify send_emails ...`
     command.
 
     '''


Test patch: