internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: sherlock-project/sherlock

Issue:
Fails when invoking script from a different directory
```
$ python sherlock/sherlock.py user123
                                              ."""-.
                                             /      \
 ____  _               _            _        |  _..--'-.
/ ___|| |__   ___ _ __| | ___   ___| |__    >.`__.-""\;"`
\___ \| '_ \ / _ \ '__| |/ _ \ / __| |/ /   / /(     ^\
 ___) | | | |  __/ |  | | (_) | (__|   <    '-`)     =|-.
|____/|_| |_|\___|_|  |_|\___/ \___|_|\_\    /`--.'--'   \ .-.
                                           .'`-._ `.\    | J /
                                          /      `--.|   \__/

[*] Removing previous file: user123.txt
[*] Checking username user123 on:
Traceback (most recent call last):
  File "sherlock/sherlock.py", line 313, in <module>
    main()
  File "sherlock/sherlock.py", line 289, in main
    results = sherlock(username, verbose=args.verbose, tor=args.tor, unique_tor=args.unique_tor)
  File "sherlock/sherlock.py", line 93, in sherlock
    raw = open("data.json", "r", encoding="utf-8")
FileNotFoundError: [Errno 2] No such file or directory: 'data.json'
```


Discussion:

Resolving patch:
diff --git a/sherlock.py b/sherlock.py
--- a/sherlock.py
+++ b/sherlock.py
@@ -90,7 +90,8 @@ def sherlock(username, verbose=False, tor=False, unique_tor=False):
     }
 
     # Load the data
-    with open("data.json", "r", encoding="utf-8") as raw:
+    data_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "data.json")
+    with open(data_file_path, "r", encoding="utf-8") as raw:
         data = json.load(raw)
 
     # Allow 1 thread for each external service, so `len(data)` threads total


Test patch:
Repository: sherlock-project/sherlock

Issue:
Version Number System
I do not think that the version number system should use the date.  There are multiple systems out there, but they are all flavors of major.minor.maintenance. This allows the version number to have some meaning to other people.

https://github.com/sdushantha/sherlock/blob/e2c4dbf1ef69db80a9c6ebf591be874686e04301/sherlock.py#L20


Discussion:
I do understand your point, but what I was thinking of was that, since we are updating the code frequently, I don't think everyone is going to have the latest version on their computer. So, if use the year.month.day format, users will get and idea if their version has been out dated or not. 

If major.minor.maintenance is the standard and has a more meaning to other people, I'm more than  happy to change it. 😊
@hoadlck Can you delete your fork of this repo?
I want to transfer the ownership of this repo to you. To do that, you first have to delete your fork.
@sdushantha  Uh... you are the owner of this repo. You cannot abandon your child! :)

I am on break now, so I had some free time to contribute. But, on 2019-01-02, the daily grind starts up again, and I am unsure how much time I will have to dedicate to this.
@hoadlck I'm not abandoning my child, I'm giving my child off to adoption :)

It is too difficult for me to maintain this repo. So I really want to give it away.
Wait, this are not the end of the conversation, right? I can still see https://github.com/TheYahya/sherlock/blob/508eb88724dbe20dedf07dc00527ad4f32c93a77/sherlock.py#L24 at the moment, yet the latest commit is already on `2019.01.03`. Maybe just have something that are regularly used like `0.0.1`, etc.
I think `semantic versioning` is more reasonable, It would be better to change it to that ☺️ 

Resolving patch:
diff --git a/sherlock.py b/sherlock.py
--- a/sherlock.py
+++ b/sherlock.py
@@ -21,7 +21,7 @@
 from torrequest import TorRequest
 
 module_name = "Sherlock: Find Usernames Across Social Networks"
-__version__ = "2018.01.04"
+__version__ = "0.1.0"
 amount=0
 
 # TODO: fix tumblr


Test patch:
Repository: sherlock-project/sherlock

Issue:
GitHub doesn't work
I would like to say it's all in the title. GitHub accounts can't be found.
GitHub doesn't work
I would like to say it's all in the title. GitHub accounts can't be found.


Discussion:

Resolving patch:
diff --git a/sherlock.py b/sherlock.py
--- a/sherlock.py
+++ b/sherlock.py
@@ -152,11 +152,11 @@ def sherlock(username, verbose=False, tor=False, unique_tor=False):
             url = net_info["url"].format(username)
             results_site["url_user"] = url
 
-            # If only the status_code is needed don't download the body
-            if net_info["errorType"] == 'status_code':
-                request_method = session.head
-            else:
-                request_method = session.get
+            request_method = session.get
+            if social_network != "GitHub":
+                # If only the status_code is needed don't download the body
+                if net_info["errorType"] == 'status_code':
+                    request_method = session.head
 
             # This future starts running the request in a new thread, doesn't block the main thread
             future = request_method(url=url, headers=headers)
@@ -380,4 +380,4 @@ def main():
                                    )
 
 if __name__ == "__main__":
-    main()
+    main()
\ No newline at end of file


Test patch:
Repository: sherlock-project/sherlock

Issue:
Sites sorting
It may be a good idea to sort the sites in sites.md and data.json alphabetically. When I'm looking for sites to add, I always have to Ctrl+F in this repo or just scroll through the file... Also when seeing the results, it's just chaos.


Discussion:
Yeah, I have same struggle. It's good to sort them.
A bit of an addition to this idea, how about adding an option to sort by alexa.com rank?
It's great @ptalmeida, but I'm afraid that the addition time to fetch ranks make it take too long.
Maybe it should be incorporated into the json data file and updated, per example, with sites_list.py.

Resolving patch:
diff --git a/site_list.py b/site_list.py
--- a/site_list.py
+++ b/site_list.py
@@ -7,6 +7,11 @@
 with open("data.json", "r", encoding="utf-8") as data_file:
     data = json.load(data_file)
 
+sorted_json_data = json.dumps(data, indent=2, sort_keys=True)
+
+with open("data.json", "w") as data_file:
+    data_file.write(sorted_json_data)
+
 with open("sites.md", "w") as site_file:
     site_file.write(f'## List Of Supported Sites ({len(data)} Sites In Total!)\n')
 


Test patch:
Repository: sherlock-project/sherlock

Issue:
[site_list.py] change numbering to reduce commit size
letting the markdown renderer do the counting lets us reduce commit size and avoide possible merge conflicts.

---

```
1.
1.
1.
```
renders to:

1.
1.
1.


Discussion:

Resolving patch:
diff --git a/site_list.py b/site_list.py
--- a/site_list.py
+++ b/site_list.py
@@ -18,11 +18,8 @@
         url_main = data.get(social_network).get("urlMain")
         pool.append((social_network, url_main))
 
-    index = 1
     for social_network, url_main in pool:
-        site_file.write(f'{index}. [{social_network}]({url_main})\n')
-        index = index + 1
-
+        site_file.write(f'1. [{social_network}]({url_main})\n')
 
 sorted_json_data = json.dumps(data, indent=2, sort_keys=True)
 


Test patch:
Repository: sherlock-project/sherlock

Issue:
Check multiple similar usernames
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->

## Checklist
<!--
Put x into all boxes (like this [x]) once you have completed what they say.
Make sure complete everything in the checklist.
-->
- [x] I'm reporting a feature request
- [x] I've checked for similar feature requests including closed ones

## Description
<!-- 
Provide a detailed description of the feature you would like Sherlock to have
-->

It's good to search similar usernames. for example when searching for `user{?}name` we should also check for: `_`, `-` and `.`
1. `username`
2. `user.name`
3. `user-name`
4. `user_name`



Discussion:
I can work on it.
when inputting sherlock {username} just add more,
example: sherlock PxINKY P_xINKY Px_INKY > pink.txt

no need to make a feature if you can just write 3 more words
Closing this as it's been PR-ed and merged already.
* https://github.com/sherlock-project/sherlock/pull/1298
Actually sorry, I realize there is no documentation on this in the README. 🤔 
I'll reopen it, later someone (probably me) can go document it.
Three more days

Sent from Yahoo Mail on Android 
 
  On Mon, Apr 4, 2022 at 2:14 PM, ***@***.***> wrote:   


when inputting sherlock {username} just add more,
example: sherlock PxINKY P_xINKY Px_INKY > pink.txt

no need to make a feature if you can just write 3 more words

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
  


Resolving patch:
diff --git a/sherlock/sherlock.py b/sherlock/sherlock.py
--- a/sherlock/sherlock.py
+++ b/sherlock/sherlock.py
@@ -551,7 +551,7 @@ def main():
     parser.add_argument("username",
                         nargs="+", metavar="USERNAMES",
                         action="store",
-                        help="One or more usernames to check with social networks."
+                        help="One or more usernames to check with social networks. Check similar usernames using {%%} (replace to '_', '-', '.')."
                         )
     parser.add_argument("--browse", "-b",
                         action="store_true", dest="browse", default=False,


Test patch:
Repository: sherlock-project/sherlock

Issue:
--timeout error
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE.
######################################################################

-->

## Checklist
<!--
Put x into all boxes (like this [x]) once you have completed what they say.
Make sure complete everything in the checklist.
-->
- [X] I'm asking a question regarding Sherlock
- [ ] My question is not a tech support question.


## Question

im having the following error: sherlock: error: argument --timeout: invalid timeout_check value: '1'
What should I write as argument of --timeout?


Discussion:
I am having the exact same problem, so if someone gets the answer tell us!

Resolving patch:
diff --git a/sherlock/sherlock.py b/sherlock/sherlock.py
--- a/sherlock/sherlock.py
+++ b/sherlock/sherlock.py
@@ -470,12 +470,14 @@ def timeout_check(value):
     NOTE:  Will raise an exception if the timeout in invalid.
     """
 
-    if value <= 0:
+    float_value = float(value)
+
+    if float_value <= 0:
         raise ArgumentTypeError(
             f"Invalid timeout value: {value}. Timeout must be a positive number."
         )
 
-    return float(value)
+    return float_value
 
 
 def handler(signal_received, frame):


Test patch:
Repository: sherlock-project/sherlock

Issue:
feat: add json schema validation
Adds a JSON Schema that validates `data.json` and `removed_sites.json`. I've already validated the existing data with it and can confirm both files pass. 👍🏾 

Prepends the following property to both files:
```json
{
  "$schema": "{path/to/json-schema}"
}
```

Removes the `rank` property from `removed_sites.json` and documentation as this is no longer relevant.

Removes the "noPeriod" property from Jimdo as this appears to serve on purpose.

---

Sorry, also a question. What is the purpose of `responseUrl`? I just realized, this property is not referenced whatsoever in the actual application. 🤔 Can this property be removed in the dataset and schema?

Later I do want to make it stricter, i.e. prevent someone from using one `errorType` with the wrong additional field, i.e.
```json
{ 
  "errorMsg": "Not Found",
  "errorType": "status_code"
} 
```

I figured I'd leave it like this for now and find out what `responseUrl` is about. 🤔 

### Related
* Closes https://github.com/sherlock-project/sherlock/issues/1336

The goal of this is to improve the experience for developers and leave less room for human-error. It also enforces that we document and provide examples of each field, unlike the current Wiki which tends to get outdated.

![image](https://user-images.githubusercontent.com/22801583/167285185-11711582-a8d1-4157-a5e9-4d590d6e71b6.png)

![image](https://user-images.githubusercontent.com/22801583/167285139-15b5d326-5813-45df-8987-dfd6f7a236ad.png)

![image](https://user-images.githubusercontent.com/22801583/167285147-e759a759-2734-4811-b972-ee75ed7dd530.png)    





Discussion:

Resolving patch:
diff --git a/sherlock/sites.py b/sherlock/sites.py
--- a/sherlock/sites.py
+++ b/sherlock/sites.py
@@ -152,6 +152,11 @@ def __init__(self, data_file_path=None):
                 raise FileNotFoundError(f"Problem while attempting to access "
                                         f"data file '{data_file_path}'."
                                         )
+        
+        try:
+            site_data.pop('$schema')
+        except:
+            pass
 
         self.sites = {}
 
diff --git a/site_list.py b/site_list.py
--- a/site_list.py
+++ b/site_list.py
@@ -5,10 +5,14 @@
 
 # Read the data.json file
 with open("sherlock/resources/data.json", "r", encoding="utf-8") as data_file:
-    data = json.load(data_file)
+    data: dict = json.load(data_file)
+
+# Removes schema-specific keywords for proper processing
+social_networks: dict = dict(data)
+social_networks.pop('$schema')
 
 # Sort the social networks in alphanumeric order
-social_networks = sorted(data.items())
+social_networks: list = sorted(social_networks.items())
 
 # Write the list of supported sites to sites.md
 with open("sites.md", "w") as site_file:


Test patch:
Repository: sherlock-project/sherlock

Issue:
SHERLOCK// ERROR "string indices must be integers, not 'str'  "
como puedo solucionar este error que me sale al querer usar sherlock, no se como solucionarlo la verdad


Discussion:

Resolving patch:
diff --git a/sherlock/sites.py b/sherlock/sites.py
--- a/sherlock/sites.py
+++ b/sherlock/sites.py
@@ -177,6 +177,8 @@ def __init__(self, data_file_path=None):
                 raise ValueError(
                     f"Problem parsing json contents at '{data_file_path}':  Missing attribute {error}."
                 )
+            except TypeError as error:
+                print(f"Encountered TypeError parsing json contents for target '{site_name}' at {data_file_path}\nSkipping target.\n")
 
         return
 


Test patch:
Repository: sherlock-project/sherlock

Issue:
False positives (Kick, LibraryThing)
Kick and LibraryThing showing F+ for `ehjtohrtjh`

_(Issue is only for tracking purposes and so I don't forget -- will pr later)_


Discussion:
I did a false positive on both of those sites when I ran Sherlock inside a Fedora container. But upon running Sherlock on my computer, I only got false positives on LibraryThing. I'm unsure as to why Kick returned a false positive, but it looks like LibraryThing has bot protection

Resolving patch:
diff --git a/sherlock/sherlock.py b/sherlock/sherlock.py
--- a/sherlock/sherlock.py
+++ b/sherlock/sherlock.py
@@ -382,11 +382,13 @@ def sherlock(
         query_status = QueryStatus.UNKNOWN
         error_context = None
 
-        # As WAFs advance and evolve, they will occasionally block Sherlock and lead to false positives
-        # and negatives. Fingerprints should be added here to filter results that fail to bypass WAFs.
-        # Fingerprints should be highly targetted. Comment at the end of each fingerprint to indicate target and date.
+        # As WAFs advance and evolve, they will occasionally block Sherlock and
+        # lead to false positives and negatives. Fingerprints should be added
+        # here to filter results that fail to bypass WAFs. Fingerprints should
+        # be highly targetted. Comment at the end of each fingerprint to
+        # indicate target and date fingerprinted.
         WAFHitMsgs = [
-            '.loading-spinner{visibility:hidden}body.no-js .challenge-running{display:none}body.dark{background-color:#222;color:#d9d9d9}body.dark a{color:#fff}body.dark a:hover{color:#ee730a;text-decoration:underline}body.dark .lds-ring div{border-color:#999 transparent transparent}body.dark .font-red{color:#b20f03}body.dark .big-button,body.dark .pow-button{background-color:#4693ff;color:#1d1d1d}body.dark #challenge-success-text{background-image:url(data:image/svg+xml;base64,', # 2024-04-08 Cloudflare
+            '.loading-spinner{visibility:hidden}body.no-js .challenge-running{display:none}body.dark{background-color:#222;color:#d9d9d9}body.dark a{color:#fff}body.dark a:hover{color:#ee730a;text-decoration:underline}body.dark .lds-ring div{border-color:#999 transparent transparent}body.dark .font-red{color:#b20f03}body.dark', # 2024-05-13 Cloudflare
             '{return l.onPageView}}),Object.defineProperty(r,"perimeterxIdentifiers",{enumerable:' # 2024-04-09 PerimeterX / Human Security
         ]
 


Test patch:
Repository: cowrie/cowrie

Issue:
Session abort when command was redirected to a file located at a non-exist directory
To reproduce the problem, execute this command: 

`echo "test" > /path/not/exists/file`

For a telnet login, the session will abort immediately.

For a SSH login, the command execution will hang there. After hitting enter, the CLI resumed. However, an `exit` command will still cause the session termination abnormally.

The problem is, in `cowrie/core/protocol.py` line 48, after noticed the command having redirection, cowrie will create the file by `fs.mkfile()`. The `mkfile()` invoked `fs.get_path()`. Since the directory doesn't exist, `get_path` raised `fs.FileNotFound` exception which was not caught properly.


Discussion:

Resolving patch:
diff --git a/cowrie/core/honeypot.py b/cowrie/core/honeypot.py
--- a/cowrie/core/honeypot.py
+++ b/cowrie/core/honeypot.py
@@ -45,9 +45,18 @@ def __init__(self, protocol, *args):
                 self.protocol.terminal.transport.session.id,
                 re.sub('[^A-Za-z0-9]', '_', self.outfile))
             perm = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
-            self.fs.mkfile(self.outfile, 0, 0, 0, stat.S_IFREG | perm)
-            with open(self.safeoutfile, 'a'):
-                self.fs.update_realfile(self.fs.getfile(self.outfile), self.safeoutfile)
+            try:
+                self.fs.mkfile(self.outfile, 0, 0, 0, stat.S_IFREG | perm)
+            except fs.FileNotFound:
+                # The outfile locates at a non-existing directory.
+                self.protocol.pp.outReceived('-bash: %s: No such file or directory\n' % self.outfile)
+                self.write = self.write_to_failed
+                self.outfile = None
+                self.safeoutfile = None
+
+            else:
+                with open(self.safeoutfile, 'a'):
+                    self.fs.update_realfile(self.fs.getfile(self.outfile), self.safeoutfile)
 
 
     def check_arguments(self, application, args):
@@ -78,10 +87,16 @@ def write_to_file(self, data):
         self.fs.update_size(self.outfile, self.writtenBytes)
 
 
+    def write_to_failed(self, data):
+        """
+        """
+        pass
+
     def start(self):
         """
         """
-        self.call()
+        if self.write != self.write_to_failed:
+            self.call()
         self.exit()
 
 


Test patch:
Repository: cowrie/cowrie

Issue:
^M for carriage return in telnet
Trying to log in via telnet to cowrie results in the following:

$telnet xx.yy.zz.aa
Trying xx.yy.zz.aa
Connected to xx.yy.zz.aa.
Escape character is '^]'.
login: icantpressenter^M^M^M^M^M^M^M^M^M^M^M^]
telnet> q
Connection closed.

It looks like some kind of control character thing. Telnet on windows "kind of works", and telnet on OS X and Linux (Debian 8.6) produce the above ^M issue.

oddly, specifying the destination port in the telnet client  (e.g "telnet host 23" ) removes this issue. Probably because control messages aren't sent.


Discussion:
I have seen the behavior too, though in my case Cowrie was running in a Docker container and connecting from outside the container produced the above ^M when pressing enter, but connecting from inside the container behaved correctly.  When connecting from outside I was not specifying a port to the Telnet client, but when inside the container I did because Cowrie was listening on port 2223.
I have just verified that I see the same behavior as you when connecting to that system with and without specifying a port.
This issue has to do with the way the Telnet client negotiates the echo, linemode, and suppress go ahead  (SGA) options and the character it the client sends when pressing enter.  When those three options are negotiated correctly, the client switches to character mode and sends a carriage return when enter is pressed.  If it is still in linemode, the client sends a newline character instead.  I do not know why the Telnet client behaves differently when a port is specified, but it is possible to get better behavior by refusing to enable SGA and linemode when authenticating.  I will submit a pull request with that change shortly.

Resolving patch:
diff --git a/cowrie/telnet/transport.py b/cowrie/telnet/transport.py
--- a/cowrie/telnet/transport.py
+++ b/cowrie/telnet/transport.py
@@ -180,15 +180,17 @@ def enableLocal(self, opt):
         if opt == ECHO:
             return True
         elif opt == SGA:
-            return True
+            return False
+            #return True
         else:
             return False
 
 
     def enableRemote(self, opt):
         if opt == LINEMODE:
-            self.transport.requestNegotiation(LINEMODE, MODE + chr(TRAPSIG))
-            return True
+            return False
+            #self.transport.requestNegotiation(LINEMODE, MODE + chr(TRAPSIG))
+            #return True
         elif opt == NAWS:
             return True
         elif opt == SGA:


Test patch:
Repository: cowrie/cowrie

Issue:
adding root:x:!password to userdb.txt doesn't exclude root/password as valid credentials
Fresh install.

I tried to exclude 'password' or 'abc123' from valid passwords for the user root

Now file looks like 
```
root:x:!root
root:x:!123456
root:x:!password
root:x:*
```

Retarted cowrie, but no way to deny login with root/password credentials

Maybe, some sort of problem with new regexp checking?


Discussion:
That used to work, maybe that stopped working with the new regex patch indeed.
Hello @supriyo-biswas 
In your patch https://github.com/micheloosterhof/cowrie/pull/763 we lost the ability to negate passwords starting with '!'. Any ideas on how to do this with regex?
Yep, experiencing the same issue here. I looked through the file and nothing stood out to me. If this doesn't get resolved I'll have more time to look at it tomorrow night.

Resolving patch:
diff --git a/cowrie/core/auth.py b/cowrie/core/auth.py
--- a/cowrie/core/auth.py
+++ b/cowrie/core/auth.py
@@ -11,6 +11,7 @@
 import json
 from os import path
 from random import randint
+from collections import OrderedDict
 
 from twisted.python import log
 
@@ -22,7 +23,7 @@ class UserDB(object):
     """
 
     def __init__(self):
-        self.userdb = {}
+        self.userdb = OrderedDict()
         self.userdb_file = '{}/userdb.txt'.format(CONFIG.get('honeypot', 'data_path'))
         self.load()
 


Test patch: