Repository: sherlock-project/sherlock

Issue:
devRant/iMGSRC.RU/ImageShack/MeetMe Not Detecting Users That Exist
Noticed while running the tests that devRant is not working right.  @TheYahya suggested during the review of #105 that we use the usernames of founders tests (as they would be expected to never delete their accounts).  But, when I tried to find  the "dfox" username, it said that it did not exist.  Yet, if I go to https://devrant.com/users/dfox, I can see his profile.

Here is the example command that demonstrates the problem:

```
python -u sherlock.py dfox --site devRant --verbose
```
Looks like @sdushantha added this site originally.


Discussion:
This appears to be a problem with the sites that redirect an attempt to view a non-existing username to the main site.  If you try to go to https://devrant.com/users/dfoxxxxxxxxx (a user name that does not exist), then we get a redirect to the https://devrant.com/ root of the site.  Yet, the "response_url" checking algorithm is only looking for the configured error URL being included in the response:

```python
        elif error_type == "response_url":
            error = net_info.get("errorUrl")
            # Checks if the redirect url is the same as the one defined in data.json
            if not error in r.url:
                print_found(social_network, url, response_time, verbose)
                write_to_file(url, f)
                exists = "yes"
                amount = amount+1
            else:
                print_not_found(social_network, response_time, verbose)
                exists = "no"
```
Since "https://devrant.com/" is in "https://devrant.com/users/dfox", it is marked as an unknown username.

Pinterest (for example) has a response URL of "https://www.pinterest.com/?show_error=true". So this does not cause the problem, as there is no overlap between a good username and the error URL.

Resolving patch:
diff --git a/sherlock.py b/sherlock.py
--- a/sherlock.py
+++ b/sherlock.py
@@ -208,13 +208,27 @@ def sherlock(username, site_data, verbose=False, tor=False, unique_tor=False, pr
                 if net_info["errorType"] == 'status_code':
                     request_method = session.head
 
+            if net_info["errorType"] == "response_url":
+                #Site forwards request to a different URL if username not
+                #found.  Disallow the redirect so we can capture the
+                #http status from the original URL request.
+                allow_redirects = False
+            else:
+                #Allow whatever redirect that the site wants to do.
+                #The final result of the request will be what is available.
+                allow_redirects = True
+
             # This future starts running the request in a new thread, doesn't block the main thread
             if proxy != None:
                 proxies = {"http": proxy, "https": proxy}
-                future = request_method(
-                    url=url, headers=headers, proxies=proxies)
+                future = request_method(url=url, headers=headers,
+                                        proxies=proxies,
+                                        allow_redirects=allow_redirects
+                                       )
             else:
-                future = request_method(url=url, headers=headers)
+                future = request_method(url=url, headers=headers,
+                                        allow_redirects=allow_redirects
+                                       )
 
             # Store future in data for access later
             net_info["request_future"] = future
@@ -290,9 +304,13 @@ def sherlock(username, site_data, verbose=False, tor=False, unique_tor=False, pr
                 exists = "no"
 
         elif error_type == "response_url":
-            error = net_info.get("errorUrl")
-            # Checks if the redirect url is the same as the one defined in data.json
-            if not error in r.url:
+            # For this detection method, we have turned off the redirect.
+            # So, there is no need to check the response URL: it will always
+            # match the request.  Instead, we will ensure that the response
+            # code indicates that the request was successful (i.e. no 404, or
+            # forward to some odd redirect).
+            if (r.status_code >= 200) and (r.status_code < 300):
+                #
                 print_found(social_network, url, response_time, verbose)
                 write_to_file(url, f)
                 exists = "yes"


Test patch:
diff --git a/tests/all.py b/tests/all.py
--- a/tests/all.py
+++ b/tests/all.py
@@ -23,7 +23,7 @@ def test_detect_true(self):
         """
 
         self.username_check(['jack'],  ['Twitter'],   exist_check=True)
-        #self.username_check(['dfox'],  ['devRant'],   exist_check=True)
+        self.username_check(['dfox'],  ['devRant'],   exist_check=True)
         self.username_check(['blue'],  ['Pinterest'], exist_check=True)
         self.username_check(['kevin'], ['Instagram'], exist_check=True)
         self.username_check(['zuck'],  ['Facebook'],  exist_check=True)
@@ -92,3 +92,51 @@ def test_detect_false_via_response_url(self):
                            )
 
         return
+
+
+class SherlockSiteCoverageTests(SherlockBaseTest):
+    def test_coverage_false_via_response_url(self):
+        """Test Username Does Not Exist Site Coverage (Via Response URL).
+
+        This test checks all sites with the "response URL" detection mechanism
+        to ensure that a Username that does not exist is reported that way.
+
+        Keyword Arguments:
+        self                   -- This object.
+
+        Return Value:
+        N/A.
+        Will trigger an assert if detection mechanism did not work as expected.
+        """
+
+        self.username_check(['noonewouldeverusethis7'],
+                            ["Pinterest", "iMGSRC.RU", "Pastebin",
+                             "WordPress", "devRant", "ImageShack", "MeetMe"
+                            ],
+                            exist_check=False
+                           )
+
+        return
+
+    def test_coverage_true_via_response_url(self):
+        """Test Username Does Exist Site Coverage (Via Response URL).
+
+        This test checks all sites with the "response URL" detection mechanism
+        to ensure that a Username that does exist is reported that way.
+
+        Keyword Arguments:
+        self                   -- This object.
+
+        Return Value:
+        N/A.
+        Will trigger an assert if detection mechanism did not work as expected.
+        """
+
+        self.username_check(['blue'],
+                            ["Pinterest", "iMGSRC.RU", "Pastebin",
+                             "WordPress", "devRant", "ImageShack", "MeetMe"
+                            ],
+                            exist_check=True
+                           )
+
+        return