Repository: gratipay/gratipay.com

Issue:
Twitter user avatars are broken
![screen shot 2014-01-22 at 4 48 16 pm](https://f.cloud.github.com/assets/688886/1979880/56562324-83b7-11e3-9114-ffdb9b933acd.png)

This is currently being discussed in [IRC](https://botbot.me/freenode/gittip/msg/10088108/).



Discussion:
Appears to be a Twitter issue: https://dev.twitter.com/discussions/25385.

Here's a script to convert `si0` to `pbs` if we need to go that route (take out the rollback):

``` sql
BEGIN;

    UPDATE elsewhere SET user_info=user_info || (
        'profile_image_url_https'=>(
            'https://pbs.' || substring(user_info->'profile_image_url_https' from 11)
        )
    ) WHERE platform='twitter';

    SELECT user_info->'profile_image_url_https' FROM elsewhere WHERE platform='twitter';

    ROLLBACK;

END;
```

Still borken.

@whit537 do all URLs in the database use `si0` currently?

@seanlinsley https://botbot.me/freenode/gittip/msg/10088765/

Is there any correlation between the 4431 users with a `pbs` URL? Are they new?

Are there any other subdomains in use?

@seanlinsley I really don't know. But I know that there is more subdomains on twtimg (like a0). 

I sugest to change @whit537 SQL and make a script that do this. (pseudo code)

```
#check image working
if get(profile_image_url_https).code in [403, 404]:
     #check if changed link works
     if get(updated_to_pbs_profile_image_url_https).code in [200, 301]
            #if works then update URL
            update_url()
```

Some of these are coming back, others aren't. MaxCDN and Bountysource are back, UkuleleRod isn't. Could be because the first two have logged in since this started.

Confirmed: MaxCDN and Bountysource are now on `pbs`, while UkuleleRod is still on `si0`. I checked a backup from last week and all three were on `si0` last week.

What's the harm in switching everyone who is `si0` to `pbs`, per https://github.com/gittip/www.gittip.com/issues/1936#issuecomment-33078357? I suppose we're assuming that all `si0`s are busted and all `pbs`s are good. We could/should verify that assumption before pulling the trigger.

``` python
#!/usr/bin/env python                                                                                       
import requests, sys                                                                                        

for i, line in enumerate(open('twimg.csv')):                                                                
    url = line.strip()                                                                                      
    response = requests.get(url)                                                                            
    if response.status_code != 200:                                                                         
        print response.status_code, url                                                                     
    sys.stdout.flush()
```

I'm running that script against 18,960 URLs. Will report back ...

Just don't do it from production :D

:-)

```
[gittip] $ grep "403 " twimg.log | wc -l
   13123
[gittip] $ grep "404 " twimg.log | wc -l
     534
[gittip] $ grep "si0" twimg.log | wc -l
   13123
[gittip] $ grep "pbs" twimg.log | wc -l
     534
[gittip] $ wc -l twimg.log
   13657 twimg.log
[gittip] $ echo 13123 534 + p | dc
13657
[gittip] $
```

The script died before reaching 18,960, not sure why. Also, why are the `pbs` ones 404 instead of 200?

Blech. This sucks.

The right ways to fix this are:
- Don't delegate profile images to accounts elsewhere (#648 #920 #1253).
- Continually update accounts elsewhere user_info (#900).

Neither of those is trivial.


Resolving patch:
diff --git a/scripts/update_user_info.py b/scripts/update_user_info.py
new file mode 100644
--- /dev/null
+++ b/scripts/update_user_info.py
@@ -0,0 +1,78 @@
+#!/usr/bin/env python
+"""This is a one-off script to update user_info for #1936.
+
+This could be generalized for #900.
+
+"""
+import os
+import time
+
+import requests
+from gittip import wireup
+from requests_oauthlib import OAuth1
+
+
+db = wireup.db()
+oauth = OAuth1( os.environ['TWITTER_CONSUMER_KEY']
+              , os.environ['TWITTER_CONSUMER_SECRET']
+              , os.environ['TWITTER_ACCESS_TOKEN']
+              , os.environ['TWITTER_ACCESS_TOKEN_SECRET']
+               )
+elsewhere = db.all("SELECT user_id FROM ELSEWHERE WHERE platform='twitter' ORDER BY id;")
+url = "https://api.twitter.com/1.1/users/lookup.json"
+
+while elsewhere:
+    batch = elsewhere[:100]
+    elsewhere = elsewhere[100:]
+    user_ids = ','.join([str(user_id) for user_id in batch])
+
+    response = requests.post(url, data={'user_id': user_ids}, auth=oauth)
+
+
+    # Log the rate-limit.
+    # ===================
+
+    nremaining = int(response.headers['X-RATE-LIMIT-REMAINING'])
+    reset = int(response.headers['X-RATE-LIMIT-RESET'])
+    print nremaining, reset, time.time()
+
+
+    if response.status_code != 200:
+
+        # Who knows what happened?
+        # ========================
+        # Supposedly we shouldn't hit 429, at least.
+
+        print response.status_code, response.text
+
+    else:
+
+        # Update!
+        # =======
+
+        users = response.json()
+
+        for user_info in users:
+
+            # flatten per upsert method in gittip/elsewhere/__init__.py
+            for k, v in user_info.items():
+                user_info[k] = unicode(v)
+
+            user_id = user_info['id']
+
+            db.run("UPDATE elsewhere SET user_info=%s WHERE user_id=%s", (user_info, user_id))
+
+            print "updated {} ({})".format(user_info['screen_name'], user_id)
+
+
+    # Stay under our rate limit.
+    # =========================
+    # We get 180 per 15 minutes for the users/lookup endpoint, per:
+    #
+    #   https://dev.twitter.com/docs/rate-limiting/1.1/limits
+
+    sleep_for = 5
+    if nremaining == 0:
+        sleep_for = reset - time.time()
+        sleep_for += 10  # Account for potential clock skew between us and Twitter.
+    time.sleep(sleep_for)


Test patch: