Repository: zulip/zulip

Issue:
Add support for setting counts in iOS push notifications
Following da8f4bc0e956e29b140fa89e963b67e3d23581df, we now have support for clearing old GCM push notifications as users read messages.  We should be able to use this same infrastructure for iOS.  It's possible that we can just grab the count from the `len(get_mobile_push_notification_ids(user_profile))` function (needs some refactoring if we do that, since it's a test function) and set `count` to that.  But testing is required, and we might still need to trigger some sort of notification on-read (otherwise, how do counts go down?).  
Add support for setting counts in iOS push notifications
Following da8f4bc0e956e29b140fa89e963b67e3d23581df, we now have support for clearing old GCM push notifications as users read messages.  We should be able to use this same infrastructure for iOS.  It's possible that we can just grab the count from the `len(get_mobile_push_notification_ids(user_profile))` function (needs some refactoring if we do that, since it's a test function) and set `count` to that.  But testing is required, and we might still need to trigger some sort of notification on-read (otherwise, how do counts go down?).  


Discussion:
Would this approach include actually clearing the push notification (e.g the message from the Notification Center) or is this just about the badge count?
This approach will actually clear the push notifications.  
For actually clearing the notifications, we have more details now in zulip/zulip-mobile#3119 . That will actually turn out to require some more work for iOS, so we might or might not ship it at the same time as we start setting badge counts. It's true that the essential information in the database for them is the same (and the same as we'll use for Android via GCM.)

I just closed https://github.com/zulip/zulip/pull/10683, which was the essentially useless seed for a possible backend change to set the badge counts.  @gnprice, I'm assuming you're driving the question of when to invest in doing the notification counts stuff for iOS.
Is there a version that has badge icon unread message counts that we can use on mobile?  I read a lot of issues about this, but no conclusions on if it will be implemented.
That's been fully implemented as works well on Android.  I'm not sure how much plumbing is required to make this work on iOS; @gnprice would know.  But probably not a ton.
Hello Greg @gnprice - would you have any insights here, or something I could try?  My users are desperate for this feature.
Hi @maltokyo -- this isn't something we've worked on recently.

One thing we'll need to figure out before we build it is what exactly the count should be a count of. To help us with that, I'd be curious to understand better how you and your users would use the badge count. When you say y'all are desperate for it: what are you missing without it? What do you find yourself trying to do, and can't do because it isn't there?

Or: is there an app or two that manages badge counts really well, and in a similar way to how you'd like to see Zulip do so? What are those apps; and how do you use their badge counts?

@zulipbot claim
Hello Greg, thank you for replying.
Currently, most of my users are on iOS, and a few on android.  They want
the same sort of unread message count that the desktop apps seem to have.
So when any stream has unread topics, then a counter for each topic would
be good.

On Wed, Sep 25, 2019 at 2:38 AM Greg Price <notifications@github.com> wrote:

> Hi @maltokyo <https://github.com/maltokyo> -- this isn't something we've
> worked on recently.
>
> One thing we'll need to figure out before we build it is what exactly the
> count should be a count of. To help us with that, I'd be curious to
> understand better how you and your users would use the badge count. When
> you say y'all are desperate for it: what are you missing without it? What
> do you find yourself trying to do, and can't do because it isn't there?
>
> Or: is there an app or two that manages badge counts really well, and in a
> similar way to how you'd like to see Zulip do so? What are those apps; and
> how do you use their badge counts?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/zulip/zulip/issues/10271?email_source=notifications&email_token=AAKS7WZF547D4L4Z5L6CH5DQLKXJTA5CNFSM4FPDIOQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QGXUQ#issuecomment-534801362>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKS7W6RESNTQA67HXKOY4LQLKXJTANCNFSM4FPDIOQQ>
> .
>

Would this approach include actually clearing the push notification (e.g the message from the Notification Center) or is this just about the badge count?
This approach will actually clear the push notifications.  
For actually clearing the notifications, we have more details now in zulip/zulip-mobile#3119 . That will actually turn out to require some more work for iOS, so we might or might not ship it at the same time as we start setting badge counts. It's true that the essential information in the database for them is the same (and the same as we'll use for Android via GCM.)

I just closed https://github.com/zulip/zulip/pull/10683, which was the essentially useless seed for a possible backend change to set the badge counts.  @gnprice, I'm assuming you're driving the question of when to invest in doing the notification counts stuff for iOS.
Is there a version that has badge icon unread message counts that we can use on mobile?  I read a lot of issues about this, but no conclusions on if it will be implemented.
That's been fully implemented as works well on Android.  I'm not sure how much plumbing is required to make this work on iOS; @gnprice would know.  But probably not a ton.
Hello Greg @gnprice - would you have any insights here, or something I could try?  My users are desperate for this feature.
Hi @maltokyo -- this isn't something we've worked on recently.

One thing we'll need to figure out before we build it is what exactly the count should be a count of. To help us with that, I'd be curious to understand better how you and your users would use the badge count. When you say y'all are desperate for it: what are you missing without it? What do you find yourself trying to do, and can't do because it isn't there?

Or: is there an app or two that manages badge counts really well, and in a similar way to how you'd like to see Zulip do so? What are those apps; and how do you use their badge counts?

@zulipbot claim
Hello Greg, thank you for replying.
Currently, most of my users are on iOS, and a few on android.  They want
the same sort of unread message count that the desktop apps seem to have.
So when any stream has unread topics, then a counter for each topic would
be good.

On Wed, Sep 25, 2019 at 2:38 AM Greg Price <notifications@github.com> wrote:

> Hi @maltokyo <https://github.com/maltokyo> -- this isn't something we've
> worked on recently.
>
> One thing we'll need to figure out before we build it is what exactly the
> count should be a count of. To help us with that, I'd be curious to
> understand better how you and your users would use the badge count. When
> you say y'all are desperate for it: what are you missing without it? What
> do you find yourself trying to do, and can't do because it isn't there?
>
> Or: is there an app or two that manages badge counts really well, and in a
> similar way to how you'd like to see Zulip do so? What are those apps; and
> how do you use their badge counts?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/zulip/zulip/issues/10271?email_source=notifications&email_token=AAKS7WZF547D4L4Z5L6CH5DQLKXJTA5CNFSM4FPDIOQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QGXUQ#issuecomment-534801362>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKS7W6RESNTQA67HXKOY4LQLKXJTANCNFSM4FPDIOQQ>
> .
>


Resolving patch:
diff --git a/zerver/lib/push_notifications.py b/zerver/lib/push_notifications.py
--- a/zerver/lib/push_notifications.py
+++ b/zerver/lib/push_notifications.py
@@ -3,7 +3,7 @@
 import logging
 import re
 import time
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence, Tuple, Union
 
 import gcm
 import lxml.html
@@ -610,6 +610,19 @@ def get_apns_alert_subtitle(message: Message) -> str:
     # For group PMs, or regular messages to a stream, just use a colon to indicate this is the sender.
     return message.sender.full_name + ":"
 
+def get_apns_badge_count(user_profile: UserProfile, read_messages_ids: Optional[Sequence[int]]=[]) -> int:
+    return UserMessage.objects.filter(
+        user_profile=user_profile
+    ).extra(
+        where=[UserMessage.where_active_push_notification()]
+    ).exclude(
+        # If we've just marked some messages as read, they're still
+        # marked as having active notifications; we'll clear that flag
+        # only after we've sent that update to the devices.  So we need
+        # to exclude them explicitly from the count.
+        message_id__in=read_messages_ids
+    ).count()
+
 def get_message_payload_apns(user_profile: UserProfile, message: Message) -> Dict[str, Any]:
     '''A `message` payload for iOS, via APNs.'''
     zulip_data = get_message_payload(user_profile, message)
@@ -625,7 +638,7 @@ def get_message_payload_apns(user_profile: UserProfile, message: Message) -> Dic
             'body': content,
         },
         'sound': 'default',
-        'badge': 0,  # TODO: set badge count in a better way
+        'badge': get_apns_badge_count(user_profile),
         'custom': {'zulip': zulip_data},
     }
     return apns_data
@@ -664,6 +677,18 @@ def get_remove_payload_gcm(
     gcm_options = {'priority': 'normal'}
     return gcm_payload, gcm_options
 
+def get_remove_payload_apns(user_profile: UserProfile, message_ids: List[int]) -> Dict[str, Any]:
+    zulip_data = get_base_payload(user_profile)
+    zulip_data.update({
+        'event': 'remove',
+        'zulip_message_ids': ','.join(str(id) for id in message_ids),
+    })
+    apns_data = {
+        'badge': get_apns_badge_count(user_profile, message_ids),
+        'custom': {'zulip': zulip_data},
+    }
+    return apns_data
+
 def handle_remove_push_notification(user_profile_id: int, message_ids: List[int]) -> None:
     """This should be called when a message that had previously had a
     mobile push executed is read.  This triggers a mobile push notifica
@@ -674,17 +699,22 @@ def handle_remove_push_notification(user_profile_id: int, message_ids: List[int]
     user_profile = get_user_profile_by_id(user_profile_id)
     message_ids = bulk_access_messages_expect_usermessage(user_profile_id, message_ids)
     gcm_payload, gcm_options = get_remove_payload_gcm(user_profile, message_ids)
+    apns_payload = get_remove_payload_apns(user_profile, message_ids)
 
     if uses_notification_bouncer():
         send_notifications_to_bouncer(user_profile_id,
-                                      {},
+                                      apns_payload,
                                       gcm_payload,
                                       gcm_options)
     else:
         android_devices = list(PushDeviceToken.objects.filter(
             user=user_profile, kind=PushDeviceToken.GCM))
+        apple_devices = list(PushDeviceToken.objects.filter(
+            user=user_profile, kind=PushDeviceToken.APNS))
         if android_devices:
             send_android_push_notification(android_devices, gcm_payload, gcm_options)
+        if apple_devices:
+            send_apple_push_notification(user_profile_id, apple_devices, apns_payload)
 
     UserMessage.objects.filter(
         user_profile_id=user_profile_id,


Test patch:
diff --git a/zerver/tests/test_push_notifications.py b/zerver/tests/test_push_notifications.py
--- a/zerver/tests/test_push_notifications.py
+++ b/zerver/tests/test_push_notifications.py
@@ -24,12 +24,14 @@
     do_delete_messages,
     do_mark_stream_messages_as_read,
     do_regenerate_api_key,
+    do_update_message_flags,
 )
 from zerver.lib.push_notifications import (
     DeviceToken,
     absolute_avatar_url,
     b64_to_hex,
     datetime_to_timestamp,
+    get_apns_badge_count,
     get_apns_client,
     get_display_recipient,
     get_message_payload_apns,
@@ -923,15 +925,28 @@ def test_send_remove_notifications_to_bouncer(self) -> None:
 
         with self.settings(PUSH_NOTIFICATION_BOUNCER_URL=True), \
                 mock.patch('zerver.lib.push_notifications'
-                           '.send_notifications_to_bouncer') as mock_send_android, \
-                mock.patch('zerver.lib.push_notifications.get_base_payload',
-                           return_value={'gcm': True}):
+                           '.send_notifications_to_bouncer') as mock_send:
             handle_remove_push_notification(user_profile.id, [message.id])
-            mock_send_android.assert_called_with(
+            mock_send.assert_called_with(
                 user_profile.id,
-                {},
                 {
-                    'gcm': True,
+                    'badge': 0,
+                    'custom': {
+                        'zulip': {
+                            'server': 'testserver',
+                            'realm_id': self.sender.realm.id,
+                            'realm_uri': 'http://zulip.testserver',
+                            'user_id': self.user_profile.id,
+                            'event': 'remove',
+                            'zulip_message_ids': str(message.id),
+                        },
+                    },
+                },
+                {
+                    'server': 'testserver',
+                    'realm_id': self.sender.realm.id,
+                    'realm_uri': 'http://zulip.testserver',
+                    'user_id': self.user_profile.id,
                     'event': 'remove',
                     'zulip_message_ids': str(message.id),
                     'zulip_message_id': message.id,
@@ -955,20 +970,41 @@ def test_non_bouncer_push_remove(self) -> None:
             PushDeviceToken.objects.filter(user=self.user_profile,
                                            kind=PushDeviceToken.GCM))
 
+        apple_devices = list(
+            PushDeviceToken.objects.filter(user=self.user_profile,
+                                           kind=PushDeviceToken.APNS))
+
         with mock.patch('zerver.lib.push_notifications'
                         '.send_android_push_notification') as mock_send_android, \
-                mock.patch('zerver.lib.push_notifications.get_base_payload',
-                           return_value={'gcm': True}):
+                mock.patch('zerver.lib.push_notifications'
+                           '.send_apple_push_notification') as mock_send_apple:
             handle_remove_push_notification(self.user_profile.id, [message.id])
             mock_send_android.assert_called_with(
                 android_devices,
                 {
-                    'gcm': True,
+                    'server': 'testserver',
+                    'realm_id': self.sender.realm.id,
+                    'realm_uri': 'http://zulip.testserver',
+                    'user_id': self.user_profile.id,
                     'event': 'remove',
                     'zulip_message_ids': str(message.id),
                     'zulip_message_id': message.id,
                 },
                 {'priority': 'normal'})
+            mock_send_apple.assert_called_with(
+                self.user_profile.id,
+                apple_devices,
+                {'badge': 0,
+                 'custom': {
+                     'zulip': {
+                         'server': 'testserver',
+                         'realm_id': self.sender.realm.id,
+                         'realm_uri': 'http://zulip.testserver',
+                         'user_id': self.user_profile.id,
+                         'event': 'remove',
+                         'zulip_message_ids': str(message.id),
+                     }
+                 }})
             user_message = UserMessage.objects.get(user_profile=self.user_profile,
                                                    message=message)
             self.assertEqual(user_message.flags.active_mobile_push_notification, False)
@@ -1150,12 +1186,39 @@ def test_modernize_apns_payload(self) -> None:
         self.assertEqual(
             modernize_apns_payload(
                 {'alert': 'Message from Hamlet',
-                 'message_ids': [3]}),
+                 'message_ids': [3],
+                 'badge': 0}),
             payload)
         self.assertEqual(
             modernize_apns_payload(payload),
             payload)
 
+    @mock.patch('zerver.lib.push_notifications.push_notifications_enabled', return_value = True)
+    def test_apns_badge_count(self, mock_push_notifications: mock.MagicMock) -> None:
+        user_profile = self.example_user('othello')
+        # Test APNs badge count for personal messages.
+        message_ids = [self.send_personal_message(self.sender,
+                                                  user_profile,
+                                                  'Content of message')
+                       for i in range(3)]
+        self.assertEqual(get_apns_badge_count(user_profile), 3)
+        # Similarly, test APNs badge count for stream mention.
+        stream = self.subscribe(user_profile, "Denmark")
+        message_ids += [self.send_stream_message(self.sender,
+                                                 stream.name,
+                                                 'Hi, @**Othello, the Moor of Venice**')
+                        for i in range(2)]
+        self.assertEqual(get_apns_badge_count(user_profile), 5)
+
+        num_messages = len(message_ids)
+        # Mark the messages as read and test whether
+        # the count decreases correctly.
+        for i, message_id in enumerate(message_ids):
+            do_update_message_flags(user_profile, get_client("website"), 'add', 'read', [message_id])
+            self.assertEqual(get_apns_badge_count(user_profile), num_messages - i - 1)
+
+        mock_push_notifications.assert_called()
+
 class TestGetAPNsPayload(PushNotificationTest):
     def test_get_message_payload_apns_personal_message(self) -> None:
         user_profile = self.example_user("othello")
@@ -1206,7 +1269,7 @@ def test_get_message_payload_apns_huddle_message(self, mock_push_notifications:
                 'body': message.content,
             },
             'sound': 'default',
-            'badge': 0,
+            'badge': 1,
             'custom': {
                 'zulip': {
                     'message_ids': [message.id],