Repository: mozmeao/basket

Issue:
Put FxA user data into SFDC in addition to SFMC
FxA users will soon need to go into SFDC instead of the data extension in SFMC into which they're going now. The timing for this will need to be coordinated with LCM and the IT Salesforce folks.


Discussion:
There are some questions I have for this move and I just wanted to record them here. The current code is here for the record:

https://github.com/mozmeao/basket/blob/1a4dc149b25c789ca7539a845f80aea08947038f/basket/news/tasks.py#L339-L351

1. We are currently receiving and storing a `service` value from FxA. Do we have a field for that in SFDC? Should we?
2. We are currently storing last-login information in SFMC. Will that move too?
3. We are currently storing information about devices in SFMC. Will that move too?
4. We currently store a language that comes from FxA. Do we need to store that separately from the current language field on the Contact, or should this update that field?
5. What should we do with the email change data we get from FxA?
6. What should we do with the account deletion data we get from FxA?

I'll add to this list if more come up. Let's discuss these on this issue.
> 
> 
> There are some questions I have for this move and I just wanted to record them here. The current code is here for the record:
> 
> https://github.com/mozmeao/basket/blob/1a4dc149b25c789ca7539a845f80aea08947038f/basket/news/tasks.py#L339-L351
> 
>     1. We are currently receiving and storing a `service` value from FxA. Do we have a field for that in SFDC? Should we?

We should. Generally product information will be stored and queried in the CDP, but service has become a vital stat about a new user, as vital as language or email addy. 
> 
>     2. We are currently storing last-login information in SFMC. Will that move too?

No. My understanding, and let's please confirm with Andrew, but data like last login will continue to feed to marketing cloud where it will be fed to the CDP. Same thing with the "Sync" logins information.

SFDC will be where we store the basic information only about each account, fxa-id, email, language, service, created date. The rest of the data will be compiled and queried in the CDP.
 
> 
>     3. We are currently storing information about devices in SFMC. Will that move too?

See above.
> 
>     4. We currently store a language that comes from FxA. Do we need to store that separately from the current language field on the Contact, or should this update that field?

I think we should store both a newsletter language and an FxA language. This is my opinion but I think it's right. 
> 
>     5. What should we do with the email change data we get from FxA?

Lisa, Andrew and I have discussed storing additional email addresses on a SFDC record. We will need to create a SFDC field and send email change data to the CRM. We want to keep all email records, but know which is the appropriate one to use for contacting a user. 
> 
>     6. What should we do with the account deletion data we get from FxA?
> 
Ideally we should delete the account. But for now, let's store a boolean flag on the CRM that defaults to false for everyone. If an account is deleted we can flip to TRUE and use that to come up with a process for purging the record from the system.

> 
> I'll add to this list if more come up. Let's discuss these on this issue.

Cool. Happy to help in any way I can. 
And a question for you @pmac : every account going into SFDC will need a token or key that we can use to power their preference center page. (This eliminates using email and therefor the login for each user). Would basket generate that or CRM?
Currently when basket adds a user to SFDC it will generate a token for them, but I believe there's also some magic on the SFDC side that will generate one if a contact is added without one. So either way every contact in SFDC should have one and so the pref center page should work normally for everyone once we get them all in there.
> > There are some questions I have for this move and I just wanted to record them here. The current code is here for the record:
> > https://github.com/mozmeao/basket/blob/1a4dc149b25c789ca7539a845f80aea08947038f/basket/news/tasks.py#L339-L351
> > 
> > ```
> > 1. We are currently receiving and storing a `service` value from FxA. Do we have a field for that in SFDC? Should we?
> > ```
> 
> We should. Generally product information will be stored and queried in the CDP, but service has become a vital stat about a new user, as vital as language or email addy.
> 
@mozbrandon Will need to create a text field called 'FirstService' for this in Sandbox. This is a one time value indicating what service they (first) came in with
> > ```
> > 2. We are currently storing last-login information in SFMC. Will that move too?
> > ```
> 
> No. My understanding, and let's please confirm with Andrew, but data like last login will continue to feed to marketing cloud where it will be fed to the CDP. Same thing with the "Sync" logins information.
> 
Correct. For now, we want to keep parallel feeds in both Marketing Cloud and now this new feed in SalesCloud. Once we have tested Lauren and teams workflows, and we have tested the BigQuery feed, we can turn off the Basket FxA feed to SFMC. @mozbrandon - No action on your part
> SFDC will be where we store the basic information only about each account, fxa-id, email, language, service, created date. The rest of the data will be compiled and queried in the CDP.
> 
> > ```
> > 3. We are currently storing information about devices in SFMC. Will that move too?
> > ```
> 
> See above.
> 
> > ```
No device data will come in. @mozbrandon - No action

> > 4. We currently store a language that comes from FxA. Do we need to store that separately from the current language field on the Contact, or should this update that field?
> > ```
> 
> I think we should store both a newsletter language and an FxA language. This is my opinion but I think it's right.
> 
@mozbrandon Will need to create a field for this in Sandbox called FxA language.
> > ```
> > 5. What should we do with the email change data we get from FxA?
> > ```
> 
> Lisa, Andrew and I have discussed storing additional email addresses on a SFDC record. We will need to create a SFDC field and send email change data to the CRM. We want to keep all email records, but know which is the appropriate one to use for contacting a user.
> 
> > ```
I talked with Lauren and we do not need the multiple email addresses right now. We will handle that setup at a later time. @mozbrandon - No action for you right now. 

> > 6. What should we do with the account deletion data we get from FxA?
> > ```
> 
> Ideally we should delete the account. But for now, let's store a boolean flag on the CRM that defaults to false for everyone. If an account is deleted we can flip to TRUE and use that to come up with a process for purging the record from the system.
> 
I agree with this approach. @mozbrandon - Will need to create the boolean 'FxA Account Deleted?' 

> > I'll add to this list if more come up. Let's discuss these on this issue.
> 
> Cool. Happy to help in any way I can.


All that sounds great to me. @mozbrandon just get me the names of the new fields and I should be able to get a PR in fairly quickly.
@pmac the following fields were created in advo:

`FirstService__c` text 
`FxA_Language__c` text 
`FxA_Account_Deleted__c` boolean

I also just realized that we currently optionally store a "Created Date" for the FxA user if it's send in the data from FxA. Do we need a separate FxA_Created_Date field for this, or are we happy with just the Contact record dates we already have?
Brandon - Please create a date field. Lets grab it now while we can. Can
always scrap it later.

On Mon, Aug 26, 2019 at 1:17 PM Paul McLanahan <notifications@github.com>
wrote:

> I also just realized that we currently optionally store a "Created Date"
> for the FxA user if it's send in the data from FxA. Do we need a separate
> FxA_Created_Date field for this, or are we happy with just the Contact
> record dates we already have?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/mozmeao/basket/issues/149?email_source=notifications&email_token=ANAFD3P57XVG4AA7E24ASM3QGQ24FA5CNFSM4HDWT3EKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FRNPQ#issuecomment-525014718>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ANAFD3KHH3EJ4NYR2WCYQPDQGQ24FANCNFSM4HDWT3EA>
> .
>


-- 
Andrew Morales
Mozilla - CRM Manager

@pmac 
`FxA_Created_Date__c` Date (not Date/Time)

Resolving patch:
diff --git a/basket/news/backends/sfdc.py b/basket/news/backends/sfdc.py
--- a/basket/news/backends/sfdc.py
+++ b/basket/news/backends/sfdc.py
@@ -77,6 +77,10 @@ def cast_lower(value):
     'amo_homepage': 'AMO_Homepage_URL__c',
     'payee_id': 'PMT_Cust_Id__c',
     'fxa_id': 'FxA_Id__c',
+    'fxa_deleted': 'FxA_Account_Deleted__c',
+    'fxa_lang': 'FxA_Language__c',
+    'fxa_service': 'FirstService__c',
+    'fxa_create_date': 'FxA_Created_Date__c',
 }
 INV_FIELD_MAP = {v: k for k, v in FIELD_MAP.items()}
 FIELD_DEFAULTS = {
diff --git a/basket/news/tasks.py b/basket/news/tasks.py
--- a/basket/news/tasks.py
+++ b/basket/news/tasks.py
@@ -234,7 +234,11 @@ def fxa_email_changed(data):
 
 @et_task
 def fxa_delete(data):
-    sfmc.upsert_row('FXA_Deleted', {'FXA_ID': data['uid']})
+    try:
+        sfdc.contact.update(f'FxA_Id__c/{data["uid"]}', {'FxA_Deleted__c': True})
+    except sfapi.SalesforceResourceNotFound:
+        # don't retry if not record with this FxA_ID
+        pass
 
 
 @et_task
@@ -244,9 +248,6 @@ def fxa_verified(data):
     email = data['email']
     fxa_id = data['uid']
     create_date = data.get('createDate')
-    if create_date:
-        create_date = datetime.fromtimestamp(create_date)
-
     locale = data.get('locale')
     subscribe = data.get('marketingOptIn')
     newsletters = data.get('newsletters')
@@ -266,7 +267,16 @@ def fxa_verified(data):
     if not lang:
         return
 
-    _update_fxa_info(email, lang, fxa_id, service, create_date)
+    user_data = {
+        'email': email,
+        'source_url': fxa_source_url(metrics),
+        'country': country,
+        'fxa_lang': lang,
+        'fxa_service': service,
+        'fxa_id': fxa_id,
+    }
+    if create_date:
+        user_data['fxa_create_date'] = iso_format_unix_timestamp(create_date)
 
     add_news = None
     if newsletters:
@@ -278,15 +288,10 @@ def fxa_verified(data):
         add_news = settings.FXA_REGISTER_NEWSLETTER
 
     if add_news:
-        upsert_user.delay(SUBSCRIBE, {
-            'email': email,
-            'lang': lang,
-            'newsletters': add_news,
-            'source_url': fxa_source_url(metrics),
-            'country': country,
-        })
-    else:
-        record_source_url(email, fxa_source_url(metrics), 'fxa-no-optin')
+        user_data['newsletters'] = add_news
+
+    upsert_contact(SUBSCRIBE, user_data,
+                   get_user_data(email=email, extra_fields=['id']))
 
 
 @et_task
@@ -337,21 +342,6 @@ def _add_fxa_activity(data):
     })
 
 
-def _update_fxa_info(email, lang, fxa_id, service, create_date=None):
-    # leaving here because easier to test
-    try:
-        apply_updates('Firefox_Account_ID', {
-            'EMAIL_ADDRESS_': email,
-            'CREATED_DATE_': gmttime(create_date),
-            'FXA_ID': fxa_id,
-            'FXA_LANGUAGE_ISO2': lang,
-            'SERVICE': service,
-        })
-    except NewsletterException as e:
-        # don't report these errors to sentry until retries exhausted
-        raise RetryTask(str(e))
-
-
 @et_task
 def update_get_involved(interest_id, lang, name, email, country, email_format,
                         subscribe, message, source_url):
@@ -460,6 +450,10 @@ def upsert_contact(api_call_type, data, user_data):
     if user_data is None:
         # no user found. create new one.
         update_data['token'] = generate_token()
+        if 'fxa_lang' in update_data:
+            # set the main lang if we're creating a new user from FxA data
+            update_data.setdefault('lang', update_data['fxa_lang'])
+
         if settings.MAINTENANCE_MODE:
             sfdc_add_update.delay(update_data)
         else:


Test patch:
diff --git a/basket/news/tests/test_tasks.py b/basket/news/tests/test_tasks.py
--- a/basket/news/tests/test_tasks.py
+++ b/basket/news/tests/test_tasks.py
@@ -958,11 +958,10 @@ def test_firefox_ios_tablet(self):
         self.assertEqual(record['DEVICE_TYPE'], 'T')
 
 
-@patch('basket.news.tasks._update_fxa_info')
-@patch('basket.news.tasks.upsert_user')
+@patch('basket.news.tasks.get_user_data', return_value=None)
+@patch('basket.news.tasks.upsert_contact')
 class FxAVerifiedTests(TestCase):
-    @patch('basket.news.tasks.sfmc')
-    def test_no_subscribe(self, sfmc_mock, upsert_mock, fxa_info_mock):
+    def test_no_subscribe(self, upsert_mock, gud_mock):
         data = {
             'email': 'thedude@example.com',
             'uid': 'the-fxa-id',
@@ -970,10 +969,16 @@ def test_no_subscribe(self, sfmc_mock, upsert_mock, fxa_info_mock):
             'marketingOptIn': False,
         }
         fxa_verified(data)
-        upsert_mock.delay.assert_not_called()
-        fxa_info_mock.assert_called_with(data['email'], 'en-US', data['uid'], '', None)
+        upsert_mock.assert_called_once_with(SUBSCRIBE, {
+            'email': data['email'],
+            'source_url': settings.FXA_REGISTER_SOURCE_URL,
+            'country': '',
+            'fxa_lang': 'en-US',
+            'fxa_service': '',
+            'fxa_id': 'the-fxa-id',
+        }, None)
 
-    def test_with_subscribe(self, upsert_mock, fxa_info_mock):
+    def test_with_subscribe(self, upsert_mock, gud_mock):
         data = {
             'email': 'thedude@example.com',
             'uid': 'the-fxa-id',
@@ -982,16 +987,17 @@ def test_with_subscribe(self, upsert_mock, fxa_info_mock):
             'service': 'sync',
         }
         fxa_verified(data)
-        upsert_mock.delay.assert_called_once_with(SUBSCRIBE, {
+        upsert_mock.assert_called_once_with(SUBSCRIBE, {
             'email': data['email'],
-            'lang': 'en-US',
             'newsletters': settings.FXA_REGISTER_NEWSLETTER,
             'source_url': settings.FXA_REGISTER_SOURCE_URL,
             'country': '',
-        })
-        fxa_info_mock.assert_called_with(data['email'], 'en-US', data['uid'], 'sync', None)
+            'fxa_lang': 'en-US',
+            'fxa_service': 'sync',
+            'fxa_id': 'the-fxa-id',
+        }, None)
 
-    def test_with_newsletters(self, upsert_mock, fxa_info_mock):
+    def test_with_newsletters(self, upsert_mock, gud_mock):
         data = {
             'email': 'thedude@example.com',
             'uid': 'the-fxa-id',
@@ -1001,16 +1007,17 @@ def test_with_newsletters(self, upsert_mock, fxa_info_mock):
             'service': 'sync',
         }
         fxa_verified(data)
-        upsert_mock.delay.assert_called_once_with(SUBSCRIBE, {
+        upsert_mock.assert_called_once_with(SUBSCRIBE, {
             'email': data['email'],
-            'lang': 'en-US',
             'newsletters': 'test-pilot,take-action-for-the-internet,' + settings.FXA_REGISTER_NEWSLETTER,
             'source_url': settings.FXA_REGISTER_SOURCE_URL,
             'country': '',
-        })
-        fxa_info_mock.assert_called_with(data['email'], 'en-US', data['uid'], 'sync', None)
+            'fxa_lang': 'en-US',
+            'fxa_service': 'sync',
+            'fxa_id': 'the-fxa-id',
+        }, None)
 
-    def test_with_subscribe_and_metrics(self, upsert_mock, fxa_info_mock):
+    def test_with_subscribe_and_metrics(self, upsert_mock, gud_mock):
         data = {
             'email': 'thedude@example.com',
             'uid': 'the-fxa-id',
@@ -1024,27 +1031,34 @@ def test_with_subscribe_and_metrics(self, upsert_mock, fxa_info_mock):
             'countryCode': 'DE',
         }
         fxa_verified(data)
-        upsert_mock.delay.assert_called_with(SUBSCRIBE, {
+        upsert_mock.assert_called_with(SUBSCRIBE, {
             'email': data['email'],
-            'lang': 'en-US',
             'newsletters': settings.FXA_REGISTER_NEWSLETTER,
             'source_url': settings.FXA_REGISTER_SOURCE_URL + '?utm_campaign=bowling',
             'country': 'DE',
-        })
-        fxa_info_mock.assert_called_with(data['email'], 'en-US', data['uid'], 'monitor', None)
+            'fxa_lang': 'en-US',
+            'fxa_service': 'monitor',
+            'fxa_id': 'the-fxa-id',
+        }, None)
 
-    @patch('basket.news.tasks.sfmc')
-    def test_with_createDate(self, sfmc_mock, upsert_mock, fxa_info_mock):
-        create_date_float = 1526996035.498
-        create_date = datetime.fromtimestamp(create_date_float)
+    def test_with_createDate(self, upsert_mock, gud_mock):
         data = {
-            'createDate': create_date_float,
+            'createDate': 1526996035.498,
             'email': 'thedude@example.com',
             'uid': 'the-fxa-id',
-            'locale': 'en-US,en'
+            'locale': 'en-US,en',
+            'service': 'sync',
         }
         fxa_verified(data)
-        fxa_info_mock.assert_called_with(data['email'], 'en-US', data['uid'], '', create_date)
+        upsert_mock.assert_called_with(SUBSCRIBE, {
+            'email': data['email'],
+            'source_url': settings.FXA_REGISTER_SOURCE_URL,
+            'country': '',
+            'fxa_create_date': '2018-05-22T13:33:55.498000',
+            'fxa_id': 'the-fxa-id',
+            'fxa_lang': 'en-US',
+            'fxa_service': 'sync',
+        }, None)
 
 
 @patch('basket.news.tasks.upsert_user')