Repository: sherlock-project/sherlock

Issue:
Ranking using site_list.py only works from certain countries
Hi,

For some strange reason `site_list.py` seems to be working for some people and not for others. When I run `python3 site_list.py -r`, to update the ranking, all the ranks get set to 0 and I get errors about the XML data that is returned by data.alexa.com. This is because all I am getting is this:
![Screenshot 2020-05-17 at 07 53 41](https://user-images.githubusercontent.com/27065646/82136986-dabf6b80-9813-11ea-938c-5efa1e520bc0.png)



But when I use a VPN and change my location from Norway to Dallas, which is a city in the US, I seem to get the expected response:
![updated](https://user-images.githubusercontent.com/27065646/82137013-10645480-9814-11ea-9840-e2104451a317.png)

From this, we can understand that this API seems to not be working properly in certain countries.

This issue has been seen here:
https://github.com/matomo-org/matomo/issues/13427
https://stackoverflow.com/questions/3676376/fetching-alexa-data#comment87619041_6224304


I personally think, if the ranking does not work for everyone, then it would be best to remove it.


Discussion:
If we remove this script's access to the Alexa API, then we will have to remove the aspects of Sherlock that use the site ranking.  Is that OK?

I am thinking that if we automate the invocation of this script using GitHub actions, then no one will have to run it on their own computer any more.  Since the GitHub Actions for Sherlock appear to be running in the US, then it will work reliably.

While the script does generally work for me, it still flakes out on some sites.  I think it is because Sherlock has enough sites that it is exceeding the API limits.
> If we remove this script's access to the Alexa API, then we will have to remove the aspects of Sherlock that use the site ranking. Is that OK?

Yeah I personally think that doing so would be fine because like you mentioned, its flakes out on some sites and that might be because its exceeding the API limits.

So, removing the site ranking would be the best thing to do as it is not stable.
It seems like it better to remove the site ranking because I tried running `site_list.py` using a VPN, where my location was Los Angeles and I keep getting this response:
```xml
<!-- Need more Alexa data?  Find our APIs here: https://aws.amazon.com/alexa/ -->
```

There is rate limit

Resolving patch:
diff --git a/sherlock/sherlock.py b/sherlock/sherlock.py
--- a/sherlock/sherlock.py
+++ b/sherlock/sherlock.py
@@ -438,9 +438,6 @@ def main():
                         action="store_true",  dest="verbose", default=False,
                         help="Display extra debugging information and metrics."
                         )
-    parser.add_argument("--rank", "-r",
-                        action="store_true", dest="rank", default=False,
-                        help="Present websites ordered by their Alexa.com global rank in popularity.")
     parser.add_argument("--folderoutput", "-fo", dest="folderoutput",
                         help="If using multiple usernames, the output of the results will be saved to this folder."
                         )
@@ -557,15 +554,6 @@ def main():
                 f"Error: Desired sites not found: {', '.join(site_missing)}.")
             sys.exit(1)
 
-    if args.rank:
-        # Sort data by rank
-        site_dataCpy = dict(site_data)
-        ranked_sites = sorted(site_data, key=lambda k: ("rank" not in k, site_data[k].get("rank", sys.maxsize)))
-        site_data = {}
-        for site in ranked_sites:
-            site_data[site] = site_dataCpy.get(site)
-
-
     #Create notify object for query results.
     query_notify = QueryNotifyPrint(result=None,
                                     verbose=args.verbose,
diff --git a/sherlock/sites.py b/sherlock/sites.py
--- a/sherlock/sites.py
+++ b/sherlock/sites.py
@@ -11,9 +11,8 @@
 
 
 class SiteInformation():
-    def __init__(self, name, url_home, url_username_format, popularity_rank,
-                 username_claimed, username_unclaimed,
-                 information):
+    def __init__(self, name, url_home, url_username_format, username_claimed,
+                 username_unclaimed, information):
         """Create Site Information Object.
 
         Contains information about a specific web site.
@@ -32,10 +31,6 @@ def __init__(self, name, url_home, url_username_format, popularity_rank,
                                          usernames would show up under the
                                          "https://somesite.com/users/" area of
                                          the web site.
-        popularity_rank        -- Integer indicating popularity of site.
-                                  In general, smaller numbers mean more
-                                  popular ("0" or None means ranking
-                                  information not available).
         username_claimed       -- String containing username which is known
                                   to be claimed on web site.
         username_unclaimed     -- String containing username which is known
@@ -58,11 +53,6 @@ def __init__(self, name, url_home, url_username_format, popularity_rank,
         self.url_home            = url_home
         self.url_username_format = url_username_format
 
-        if (popularity_rank is None) or (popularity_rank == 0):
-            #We do not know the popularity, so make site go to bottom of list.
-            popularity_rank = sys.maxsize
-        self.popularity_rank     = popularity_rank
-
         self.username_claimed    = username_claimed
         self.username_unclaimed  = username_unclaimed
         self.information         = information
@@ -118,22 +108,19 @@ def __init__(self, data_file_path=None):
         """
 
         if data_file_path is None:
-            #Use internal default.
-            data_file_path = \
-                os.path.join(os.path.dirname(os.path.realpath(__file__)),
-                             "resources/data.json"
-                            )
-
-        #Ensure that specified data file has correct extension.
-        if ".json" != data_file_path[-5:].lower():
+            # The default data file is the live data.json which is in the GitHub repo. The reason why we are using
+            # this instead of the local one is so that the user has the most up to date data. This prevents
+            # users from creating issue about false positives which has already been fixed or having outdated data
+            data_file_path = "https://raw.githubusercontent.com/sherlock-project/sherlock/master/sherlock/resources/data.json"
+
+        # Ensure that specified data file has correct extension.
+        if not data_file_path.lower().endswith(".json"):
             raise FileNotFoundError(f"Incorrect JSON file extension for "
                                     f"data file '{data_file_path}'."
                                    )
 
-        if ( ("http://"  == data_file_path[:7].lower()) or
-             ("https://" == data_file_path[:8].lower())
-           ):
-            #Reference is to a URL.
+        if "http://"  == data_file_path[:7].lower() or "https://" == data_file_path[:8].lower():
+            # Reference is to a URL.
             try:
                 response = requests.get(url=data_file_path)
             except Exception as error:
@@ -172,14 +159,11 @@ def __init__(self, data_file_path=None):
         #Add all of site information from the json file to internal site list.
         for site_name in site_data:
             try:
-                #If popularity unknown, make site be at bottom of list.
-                popularity_rank = site_data[site_name].get("rank", sys.maxsize)
 
                 self.sites[site_name] = \
                     SiteInformation(site_name,
                                     site_data[site_name]["urlMain"],
                                     site_data[site_name]["url"],
-                                    popularity_rank,
                                     site_data[site_name]["username_claimed"],
                                     site_data[site_name]["username_unclaimed"],
                                     site_data[site_name]
@@ -192,32 +176,17 @@ def __init__(self, data_file_path=None):
 
         return
 
-    def site_name_list(self, popularity_rank=False):
+    def site_name_list(self):
         """Get Site Name List.
 
         Keyword Arguments:
         self                   -- This object.
-        popularity_rank        -- Boolean indicating if list should be sorted
-                                  by popularity rank.
-                                  Default value is False.
-                                  NOTE:  List is sorted in ascending
-                                         alphabetical order is popularity rank
-                                         is not requested.
 
         Return Value:
         List of strings containing names of sites.
         """
 
-        if popularity_rank:
-            #Sort in ascending popularity rank order.
-            site_rank_name = \
-                sorted([(site.popularity_rank,site.name) for site in self],
-                       key=operator.itemgetter(0)
-                      )
-            site_names = [name for _,name in site_rank_name]
-        else:
-            #Sort in ascending alphabetical order.
-            site_names = sorted([site.name for site in self], key=str.lower)
+        site_names = sorted([site.name for site in self], key=str.lower)
 
         return site_names
 
diff --git a/site_list.py b/site_list.py
--- a/site_list.py
+++ b/site_list.py
@@ -1,40 +1,12 @@
 """Sherlock: Supported Site Listing
-This module generates the listing of supported sites.
+This module generates the listing of supported sites
+which can be found in sites.md
+It also organizes all the sites in alphanumeric order
 """
 import json
-import sys
-import requests
-import threading
-import xml.etree.ElementTree as ET
-from datetime import datetime
-from argparse import ArgumentParser, RawDescriptionHelpFormatter
 
 pool = list()
 
-def get_rank(domain_to_query, dest):
-
-    #Retrieve ranking data via alexa API
-    url = f"http://data.alexa.com/data?cli=10&url={domain_to_query}"
-    xml_data = requests.get(url).text
-    root = ET.fromstring(xml_data)
-    try:
-        #Get ranking for this site.
-        dest['rank'] = int(root.find(".//REACH").attrib["RANK"])
-    except:
-        #We did not find the rank for some reason.
-        print(f"Error retrieving rank information for '{domain_to_query}'")
-        print(f"     Returned XML is |{xml_data}|")
-
-    return
-
-parser = ArgumentParser(formatter_class=RawDescriptionHelpFormatter
-                        )
-parser.add_argument("--rank","-r",
-                    action="store_true",  dest="rank", default=False,
-                    help="Update all website ranks (not recommended)."
-                    )
-args = parser.parse_args()
-
 with open("sherlock/resources/data.json", "r", encoding="utf-8") as data_file:
     data = json.load(data_file)
 
@@ -44,30 +16,17 @@ def get_rank(domain_to_query, dest):
 
     for social_network in data:
         url_main = data.get(social_network).get("urlMain")
-        data.get(social_network)["rank"] = 0
-        if args.rank:
-            th = threading.Thread(target=get_rank, args=(url_main, data.get(social_network)))
-        else:
-            th = None
-        pool.append((social_network, url_main, th))
-        if args.rank:
-            th.start()
+        pool.append((social_network, url_main))
 
     index = 1
-    for social_network, url_main, th in pool:
-        if args.rank:
-            th.join()
+    for social_network, url_main in pool:
         site_file.write(f'{index}. [{social_network}]({url_main})\n')
-        sys.stdout.write("\r{0}".format(f"Updated {index} out of {data_length} entries"))
-        sys.stdout.flush()
         index = index + 1
 
-    if args.rank:
-        site_file.write(f'\nAlexa.com rank data fetched at ({datetime.utcnow()} UTC)\n')
 
 sorted_json_data = json.dumps(data, indent=2, sort_keys=True)
 
 with open("sherlock/resources/data.json", "w") as data_file:
     data_file.write(sorted_json_data)
 
-print("\nFinished updating supported site listing!")
+print("Finished updating supported site listing!")


Test patch: