Repository: mozmeao/basket

Issue:
Move FxA Activity data from a view to the queue
The Queue of events we get from FxA includes `login` events that have the same info as the posts we get from FxA's basket proxy to `/fxa-activity/`. We should consume these events instead of having FxA send us these POST requests.

**NOTE**: These posts are not idempotent, so we need to take special care in the transition to be sure we don't duplicate or drop any data.

See the discussion in https://github.com/mozmeao/basket/pull/35#pullrequestreview-61304053 for reference.

**NOTE 2**: We need to be sure we're handling the extra subscriptions for unique welcome emails implemented in the basket-proxy:

https://github.com/mozilla/fxa-basket-proxy/blob/13ed0f5206f9150566135d3a0973ac358cf1c0f7/lib/events/index.js#L96-L111

https://github.com/mozilla/fxa-basket-proxy/blob/13ed0f5206f9150566135d3a0973ac358cf1c0f7/lib/config.js#L57-L69


Discussion:
cc @rfk
I'm assuming that calls to `/fxa-activity/` are not idempotent, but is that true in practice?  I have a vague recollection of there being an issue where we were missing some login events, and one of the theories was that they might be so close together that they were generating duplicate rows in the underlying data extension.
I'm pretty sure that's true. We just record whatever update we get for this one. I'm not sure it would matter much in the long run if there were dupes, but it's best kept to a minimum I'm sure.
Just spoke to @rfk and we've decided that the best path forward for doing this data cut-over is to use the timestamps on the queue messages. The procedure should be basically as follows:

1. Decide on a timestamp value before which everything will go over the basket-proxy, and after which everything will happen over the queue
2. Code that timestamp into the fxa-basket-proxy so that it will stop sending basket requests after that timestamp
3. Code that timestamp into basket so that it will start accepting these queue messages after that timestamp.
4. Profit

We'll need to get the work from 2 and 3 done first with the timestamp configurable on both sides, then choose said timestamp and deploy. I'll go ahead and get started on supporting these messages and only doing so after a specific value in the timestamp field.
Note: timestamp key in the message will be `ts` and will be a float in seconds since unix epoch.

see https://github.com/mozilla/fxa-auth-server/pull/2168/
I think the timestamps *should* be live in the event stream as of now, are you able to confirm from your end?

Resolving patch:
diff --git a/basket/news/management/commands/process_fxa_queue.py b/basket/news/management/commands/process_fxa_queue.py
--- a/basket/news/management/commands/process_fxa_queue.py
+++ b/basket/news/management/commands/process_fxa_queue.py
@@ -12,12 +12,27 @@
 from django_statsd.clients import statsd
 from raven.contrib.django.raven_compat.models import client as sentry_client
 
-from basket.news.tasks import fxa_delete, fxa_verified
+from basket.news.tasks import fxa_delete, fxa_login, fxa_verified
+
+
+# TODO remove this after the cutover
+class FxATSProxyTask(object):
+    """Fake task that will only fire the real task after timestamp"""
+    def __init__(self, task, timestamp):
+        self.task = task
+        self.ts = timestamp
+
+    def delay(self, data):
+        if not self.ts or data['ts'] < self.ts:
+            return
+
+        self.task.delay(data)
 
 
 FXA_EVENT_TYPES = {
     'delete': fxa_delete,
     'verified': fxa_verified,
+    'login': FxATSProxyTask(fxa_login, settings.FXA_LOGIN_CUTOVER_TIMESTAMP),
 }
 
 
diff --git a/basket/news/tasks.py b/basket/news/tasks.py
--- a/basket/news/tasks.py
+++ b/basket/news/tasks.py
@@ -203,6 +203,15 @@ def gmttime():
     return formatdate(timeval=stamp, localtime=False, usegmt=True)
 
 
+def fxa_source_url(metrics):
+    source_url = settings.FXA_REGISTER_SOURCE_URL
+    query = {k: v for k, v in metrics.items() if k.startswith('utm_')}
+    if query:
+        source_url = '?'.join((source_url, urlencode(query)))
+
+    return source_url
+
+
 @et_task
 def fxa_delete(data):
     sfmc.upsert_row('FXA_Deleted', {'FXA_ID': data['uid']})
@@ -229,21 +238,45 @@ def fxa_verified(data):
     _update_fxa_info(email, lang, fxa_id)
 
     if subscribe:
-        source_url = settings.FXA_REGISTER_SOURCE_URL
-        query = {k: v for k, v in metrics.items() if k.startswith('utm_')}
-        if query:
-            source_url = '?'.join((source_url, urlencode(query)))
-
         upsert_user.delay(SUBSCRIBE, {
             'email': email,
             'lang': lang,
             'newsletters': settings.FXA_REGISTER_NEWSLETTER,
-            'source_url': source_url,
+            'source_url': fxa_source_url(metrics),
+        })
+
+
+@et_task
+def fxa_login(data):
+    email = data['email']
+    # if we're not using the sandbox ignore testing domains
+    if email_is_testing(email):
+        return
+
+    new_data = {
+        'user_agent': data['userAgent'],
+        'fxa_id': data['uid'],
+        'first_device': data['deviceCount'] == 1,
+    }
+    _add_fxa_activity(new_data)
+
+    metrics = data.get('metricsContext', {})
+    newsletter = settings.FXA_LOGIN_CAMPAIGNS.get(metrics.get('utm_campaign'))
+    if newsletter:
+        upsert_user.delay(SUBSCRIBE, {
+            'email': email,
+            'newsletters': newsletter,
+            'source_url': fxa_source_url(metrics),
         })
 
 
 @et_task
 def add_fxa_activity(data):
+    # TODO delete after fxa_activity view is decomissioned
+    _add_fxa_activity(data)
+
+
+def _add_fxa_activity(data):
     user_agent = user_agents.parse(data['user_agent'])
     device_type = 'D'
     if user_agent.is_mobile:
diff --git a/basket/settings.py b/basket/settings.py
--- a/basket/settings.py
+++ b/basket/settings.py
@@ -346,6 +346,17 @@ def get_default_gateway_linux():
 
 FXA_REGISTER_NEWSLETTER = config('FXA_REGISTER_NEWSLETTER', default='firefox-accounts-journey')
 FXA_REGISTER_SOURCE_URL = config('FXA_REGISTER_SOURCE_URL', default='https://accounts.firefox.com/')
+# TODO remove this after the cutover
+FXA_LOGIN_CUTOVER_TIMESTAMP = config('FXA_LOGIN_CUTOVER_TIMESTAMP', default='0', cast=int)
+# TODO move this to the DB
+FXA_LOGIN_CAMPAIGNS = {
+    'fxa-embedded-form-moz': 'mozilla-welcome',
+    'fxa-embedded-form-fx': 'firefox-welcome',
+    'membership-idealo': 'member-idealo',
+    'membership-comm': 'member-comm',
+    'membership-tech': 'member-tech',
+    'membership-tk': 'member-tk',
+}
 
 OIDC_ENABLE = config('OIDC_ENABLE', default=False, cast=bool)
 if OIDC_ENABLE:


Test patch:
diff --git a/basket/news/tests/test__utils.py b/basket/news/tests/test__utils.py
--- a/basket/news/tests/test__utils.py
+++ b/basket/news/tests/test__utils.py
@@ -1,8 +1,9 @@
 # -*- coding: utf8 -*-
 
 from django.test import TestCase
-from mock import patch
+from mock import Mock, patch
 
+from basket.news.management.commands.process_fxa_queue import FxATSProxyTask
 from basket.news.models import BlockedEmail
 from basket.news.utils import (
     email_block_list_cache,
@@ -17,6 +18,52 @@
 )
 
 
+class FxATSProxyTaskTests(TestCase):
+    def test_no_call_before_timestamp(self):
+        """Should not call task if timestamp is less than desired"""
+        task = Mock()
+        data = {
+            'ts': 1231.123,
+            'dude': 'abiding',
+        }
+        t = FxATSProxyTask(task, 1234)
+        t.delay(data)
+        task.delay.assert_not_called()
+
+    def test_no_call_timestamp_disabled(self):
+        """Should not call task before timestamp is configured"""
+        task = Mock()
+        data = {
+            'ts': 0,
+            'dude': 'abiding',
+        }
+        t = FxATSProxyTask(task, 1234)
+        t.delay(data)
+        task.delay.assert_not_called()
+
+    def test_call_after_timestamp(self):
+        """Should call task after the desired timestamp"""
+        task = Mock()
+        data = {
+            'ts': 1234.123,
+            'dude': 'abiding',
+        }
+        t = FxATSProxyTask(task, 1234)
+        t.delay(data)
+        task.delay.assert_called_with(data)
+
+    def test_call_exatly_at_timestamp(self):
+        """Should call task exactly at the desired timestamp"""
+        task = Mock()
+        data = {
+            'ts': 1234.000,
+            'dude': 'abiding',
+        }
+        t = FxATSProxyTask(task, 1234)
+        t.delay(data)
+        task.delay.assert_called_with(data)
+
+
 class ParsePhoneNumberTests(TestCase):
     def test_valid_numbers(self):
         assert parse_phone_number('7068675309', 'us') == '+17068675309'
diff --git a/basket/news/tests/test_tasks.py b/basket/news/tests/test_tasks.py
--- a/basket/news/tests/test_tasks.py
+++ b/basket/news/tests/test_tasks.py
@@ -1,3 +1,4 @@
+from copy import deepcopy
 from urllib2 import URLError
 
 from django.conf import settings
@@ -14,6 +15,7 @@
     add_fxa_activity,
     add_sms_user,
     et_task,
+    fxa_login,
     fxa_verified,
     mogrify_message_id,
     NewsletterException,
@@ -598,3 +600,77 @@ def test_with_subscribe_and_metrics(self, upsert_mock, fxa_info_mock):
             'source_url': settings.FXA_REGISTER_SOURCE_URL + '?utm_campaign=bowling',
         })
         fxa_info_mock.assert_called_with(data['email'], 'en-US', data['uid'])
+
+
+@patch('basket.news.tasks.upsert_user')
+@patch('basket.news.tasks._add_fxa_activity')
+class FxALoginTests(TestCase):
+    # based on real data pulled from the queue
+    base_data = {
+        'deviceCount': 2,
+        'email': 'the.dude@example.com',
+        'event': 'login',
+        'metricsContext': {
+            'device_id': 'phones-ringing-dude',
+            'flowBeginTime': 1508897207639,
+            'flowCompleteSignal': 'account.signed',
+            'flowType': 'login',
+            'flow_id': 'the-dude-goes-with-the-flow-man',
+            'flow_time': 31568,
+            'time': 1508897239207,
+            'utm_campaign': 'fxa-embedded-form-fx',
+            'utm_content': 'fx-56.0.1',
+            'utm_medium': 'referral',
+            'utm_source': 'firstrun_f131',
+        },
+        'service': 'sync',
+        'ts': 1508897239.207,
+        'uid': 'the-fxa-id-for-el-dudarino',
+        'userAgent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0',
+    }
+
+    def get_data(self):
+        return deepcopy(self.base_data)
+
+    def test_fxa_login_task_with_no_utm(self, afa_mock, upsert_mock):
+        data = self.get_data()
+        del data['metricsContext']
+        data['deviceCount'] = 1
+        fxa_login(data)
+        afa_mock.assert_called_with({
+            'user_agent': data['userAgent'],
+            'fxa_id': data['uid'],
+            'first_device': True,
+        })
+        upsert_mock.delay.assert_not_called()
+
+    def test_fxa_login_task_with_utm_data(self, afa_mock, upsert_mock):
+        data = self.get_data()
+        fxa_login(data)
+        afa_mock.assert_called_with({
+            'user_agent': data['userAgent'],
+            'fxa_id': data['uid'],
+            'first_device': False,
+        })
+        upsert_mock.delay.assert_called_with(SUBSCRIBE, {
+            'email': 'the.dude@example.com',
+            'newsletters': settings.FXA_LOGIN_CAMPAIGNS['fxa-embedded-form-fx'],
+            'source_url': ANY,
+        })
+        source_url = upsert_mock.delay.call_args[0][1]['source_url']
+        assert 'utm_campaign=fxa-embedded-form-fx' in source_url
+        assert 'utm_content=fx-56.0.1' in source_url
+        assert 'utm_medium=referral' in source_url
+        assert 'utm_source=firstrun_f131' in source_url
+
+    def test_fxa_login_task_with_utm_data_no_subscribe(self, afa_mock, upsert_mock):
+        data = self.get_data()
+        # not in the FXA_LOGIN_CAMPAIGNS setting
+        data['metricsContext']['utm_campaign'] = 'nonesense'
+        fxa_login(data)
+        afa_mock.assert_called_with({
+            'user_agent': data['userAgent'],
+            'fxa_id': data['uid'],
+            'first_device': False,
+        })
+        upsert_mock.delay.assert_not_called()