internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Gallopsled/pwntools

Issue:
SSH Sock Weirdness
Not sure how this is happening, but `ssh_channel.sock` is being set to None.  However, if we change the code so that it reads:

``` python
        self.sock = parent.transport.open_session()
        print "%r->%r" % (self, self.sock)
```

We see that the ssh_channel object 0x7f5af19a2f10 printed out its socket.  Breaking into PDB at the faulting line (the access `self.sock.XXX` fails), we see that `self.sock` is now `None`.  Not sure how this is happening.

```
[+] Connecting to shell2014.picoctf.com on port 22: OK
<pwnlib.tubes.ssh.ssh_channel object at 0x7f5af198d150>-><paramiko.Channel 1 (open) window=0 -> <paramiko.Transport at 0xf2ef5d10L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>
<pwnlib.tubes.ssh.ssh_channel object at 0x7f5af19a2f10>-><paramiko.Channel 2 (open) window=0 -> <paramiko.Transport at 0xf2ef5d10L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>
> /home/user/pwntools/pwnlib/tubes/ssh.py(83)poll()
-> if block or (self.sock and self.sock.exit_status_ready()):
(Pdb) print self
<pwnlib.tubes.ssh.ssh_channel object at 0x7f5af19a2f10>
(Pdb) print self.sock
None
```



Discussion:
Could you give me a testcase to look at?

Here's an example which came up using just the doctests.

https://travis-ci.org/Gallopsled/pwntools/builds/45632362#L1040-L1065

Booooooooooooooooooooooooooooooooooooooooooooooooooooooo Travis

Evidently re-running a job clears the old output.

In any case, this is reproducible with the current unit tests.  It's an intermittent issue.


Resolving patch:
diff --git a/pwnlib/tubes/ssh.py b/pwnlib/tubes/ssh.py
--- a/pwnlib/tubes/ssh.py
+++ b/pwnlib/tubes/ssh.py
@@ -77,6 +77,24 @@ def kill(self):
 
         self.close()
 
+    def recvall(self):
+        # We subclass tubes.sock which sets self.sock to None.
+        #
+        # However, we need to wait for the return value to propagate,
+        # which may not happen by the time .close() is called by tube.recvall()
+        tmp_sock = self.sock
+
+        data = super(ssh_channel, self).recvall()
+
+        # Restore self.sock to be able to call wait()
+        self.sock = tmp_sock
+        self.wait()
+
+        # Again set self.sock to None
+        self.sock = None
+
+        return data
+
     def wait(self):
         return self.poll(block=True)
 
@@ -88,7 +106,7 @@ def poll(self, block=False):
         """
 
         if self.returncode == None:
-            if block or (self.sock and self.sock.exit_status_ready()):
+            if self.sock and (block or self.sock.exit_status_ready()):
                 self.returncode = self.sock.recv_exit_status()
 
         return self.returncode


Test patch:
Repository: Gallopsled/pwntools

Issue:
Importing shellcraft submodules doesn't work
```
$ python
>>> import pwnlib.shellcraft.arm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named arm
```



Discussion:

Resolving patch:
diff --git a/pwnlib/shellcraft/__init__.py b/pwnlib/shellcraft/__init__.py
--- a/pwnlib/shellcraft/__init__.py
+++ b/pwnlib/shellcraft/__init__.py
@@ -102,4 +102,22 @@ def __shellcodes__(self):
 tether = sys.modules[__name__]
 
 # Create the module structure
-module(__name__, '')
+shellcraft = module(__name__, '')
+
+class LazyImporter:
+    def find_module(self, fullname, path):
+        if not fullname.startswith('pwnlib.shellcraft.'):
+            return None
+
+        parts = fullname.split('.')[2:]
+        cur = shellcraft
+        for part in parts:
+            cur = getattr(cur, part, None)
+            if not isinstance(cur, ModuleType):
+                return None
+
+        return self
+
+    def load_module(self, fullname):
+        return sys.modules[fullname]
+sys.meta_path.append(LazyImporter())


Test patch:
Repository: Gallopsled/pwntools

Issue:
pwnlib.term.text throws exceptions on attribute accesses
```
>>> import pwnlib
>>> pwnlib.term.text.__lol__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pwnlib/term/text.py", line 99, in __getattr__
    init += self._fg_color(c())
  File "pwnlib/term/text.py", line 93, in c
    return self._colors[c] + bright
KeyError: ''
```



Discussion:

Resolving patch:
diff --git a/pwnlib/term/text.py b/pwnlib/term/text.py
--- a/pwnlib/term/text.py
+++ b/pwnlib/term/text.py
@@ -73,33 +73,36 @@ def f(self, s, when = None):
         return functools.partial(f, self)
 
     def __getattr__(self, desc):
-        ds = desc.replace('gray', 'bright_black').split('_')
-        init = ''
-        while ds:
-            d = ds[0]
-            try:
-                init += self._attributes[d]
-                ds.pop(0)
-            except KeyError:
-                break
-        def c():
-            bright = 0
-            c = ds.pop(0)
-            if c == 'bright':
+        try:
+            ds = desc.replace('gray', 'bright_black').split('_')
+            init = ''
+            while ds:
+                d = ds[0]
+                try:
+                    init += self._attributes[d]
+                    ds.pop(0)
+                except KeyError:
+                    break
+            def c():
+                bright = 0
                 c = ds.pop(0)
-                if self.has_bright:
-                    bright = 8
-            return self._colors[c] + bright
-        if ds:
-            if ds[0] == 'on':
-                ds.pop(0)
-                init += self._bg_color(c())
-            else:
-                init += self._fg_color(c())
-                if len(ds):
-                    assert ds.pop(0) == 'on'
+                if c == 'bright':
+                    c = ds.pop(0)
+                    if self.has_bright:
+                        bright = 8
+                return self._colors[c] + bright
+            if ds:
+                if ds[0] == 'on':
+                    ds.pop(0)
                     init += self._bg_color(c())
-        return self._decorator(desc, init)
+                else:
+                    init += self._fg_color(c())
+                    if len(ds):
+                        assert ds.pop(0) == 'on'
+                        init += self._bg_color(c())
+            return self._decorator(desc, init)
+        except (IndexError, KeyError):
+            raise AttributeError("'module' object has no attribute %r" % desc)
 
     def get(self, desc):
         return self.__getattr__(desc)


Test patch:
Repository: Gallopsled/pwntools

Issue:
Mac OS X Default Assembler Irreparably Broken
Mac OS X's default assembler has major issues in Intel syntax mode.  Specifically, backward jumps are broken.  The only relevant information that I could find is [here](https://discussions.apple.com/thread/2149747?tstart=0).

Additionally, the default assembler installed by `brew install binutils` does not include an assembler.

```
~/pwntools ❯❯❯ uname
Darwin
~/pwntools ❯❯❯ /usr/bin/as -v
Apple Inc version cctools-862, GNU assembler version 1.38
~/pwntools ❯❯❯ asm $'foo:\njmp foo'
[ERROR] There was an error running ['/usr/bin/as', '-arch', 'i386', '-o', '/var/folders/c8/3mf71lv51172mvh8yhzlpv1w008rd2/T/pwn-asm-guNHlD/step2', '/var/folders/c8/3mf71lv51172mvh8yhzlpv1w008rd2/T/pwn-asm-guNHlD/step1']:
    It had the exitcode 1.
    It had this on stdout:
    /var/folders/c8/3mf71lv51172mvh8yhzlpv1w008rd2/T/pwn-asm-guNHlD/step1:5:suffix or operands invalid for `jmp'

[ERROR] An error occurred while assembling:
    .org 0x0
    .section .shellcode,"ax"
    .intel_syntax noprefix
    foo:
    jmp foo
...
pwnlib.exception.PwnlibException: There was an error running ['/usr/bin/as', '-arch', 'i386', '-o', '/var/folders/c8/3mf71lv51172mvh8yhzlpv1w008rd2/T/pwn-asm-guNHlD/step2', '/var/folders/c8/3mf71lv51172mvh8yhzlpv1w008rd2/T/pwn-asm-guNHlD/step1']:
It had the exitcode 1.
It had this on stdout:
/var/folders/c8/3mf71lv51172mvh8yhzlpv1w008rd2/T/pwn-asm-guNHlD/step1:5:suffix or operands invalid for `jmp'
```



Discussion:
Note that even if the user has installed an alternate assembler, we currently have special cases for the OS X default assembler that would need to be removed before it can work.

More evidence of it being pretty busted, Gynvael none the less http://gynvael.coldwind.pl/?id=96


Resolving patch:
diff --git a/pwnlib/asm.py b/pwnlib/asm.py
--- a/pwnlib/asm.py
+++ b/pwnlib/asm.py
@@ -124,11 +124,9 @@ def _assembler():
         'little': '-EL'
     }[context.endianness]
 
-    osx = (platform.system() == 'Darwin')
-
     assemblers = {
-        'i386'   : [gas, '--32'] if not osx else [gas, '-arch', 'i386'],
-        'amd64'  : [gas, '--64'] if not osx else [gas, '-arch', 'x86_64'],
+        'i386'   : [gas, '--32'],
+        'amd64'  : [gas, '--64'],
 
         # Most architectures accept -EL or -EB
         'thumb'  : [gas, '-mthumb', E],
@@ -149,7 +147,6 @@ def _assembler():
 
     return assemblers.get(context.arch, [gas])
 
-
 def _objcopy():
     return [which_binutils('objcopy')]
 
@@ -335,10 +332,8 @@ def asm(shellcode, vma = 0, **kwargs):
     result = ''
 
     with context.local(**kwargs):
-        osx = (platform.system() == 'Darwin')
-
         assembler = _assembler()
-        objcopy   = _objcopy() + ['-j', '.shellcode' if not osx else '*.shellcode*', '-Obinary']
+        objcopy   = _objcopy() + ['-j', '.shellcode', '-Obinary']
         code      = '.org %#x\n' % vma
         code      += _arch_header()
         code      += cpp(shellcode)


Test patch:
Repository: Gallopsled/pwntools

Issue:
`pwnlib.util.fiddling.hexdump()` is slow for some input
Looks like it has to do with de Bruijn sequence detection.  Compare:

```
hexdump(randoms(1000))
```

and

```
hexdump('A' * 1000)
```



Discussion:
Do you think there's a way to do the detection more quickly?  I'm not opposed to ripping that out if it can't be done, you seem to be better at optimizing than I am.

I'm not the expert on de Bruijn sequences... @IdolfHatler.


Resolving patch:
diff --git a/pwnlib/util/fiddling.py b/pwnlib/util/fiddling.py
--- a/pwnlib/util/fiddling.py
+++ b/pwnlib/util/fiddling.py
@@ -540,7 +540,7 @@ def style_byte(b):
     for line, chunk in enumerate(lists.group(width, s)):
         # If this chunk is the same as the last unique chunk,
         # use a '*' instead.
-        if skip and (last_unique == chunk or sequential_lines(last_unique, chunk)):
+        if skip and last_unique == chunk:
             last_unique = chunk
             if not skipping:
                 yield '*'


Test patch:
Repository: Gallopsled/pwntools

Issue:
cyclic broken
$ cyclic -o 0x61616161
Traceback (most recent call last):
  File "/usr/local/bin/cyclic", line 9, in <module>
    load_entry_point('pwntools==2.2', 'console_scripts', 'cyclic')()
  File "/home/freaken/git/pwntools/pwnlib/commandline/cyclic.py", line 49, in main
    pat = packing.pack(int(pat[2:], 16), subsize*8, 'little', 'unsigned')
  File "/home/freaken/git/pwntools/pwnlib/util/packing.py", line 94, in pack
    raise ValueError("pack(): sign must be either True or False (got %r)" % sign)
ValueError: pack(): sign must be either True or False (got 'unsigned')



Discussion:

Resolving patch:
diff --git a/pwnlib/util/packing.py b/pwnlib/util/packing.py
--- a/pwnlib/util/packing.py
+++ b/pwnlib/util/packing.py
@@ -81,11 +81,17 @@ def pack(number, word_size = None, endianness = None, sign = None, **kwargs):
         >>> pack(0x0102030405, 'all', 'little', True)
         '\\x05\\x04\\x03\\x02\\x01'
 """
+    kwargs.setdefault('endianness', endianness)
+    kwargs.setdefault('sign', sign)
+
+    if word_size != 'all':
+        kwargs.setdefault('word_size', word_size)
+
     with context.local(**kwargs):
         # Lookup in context if not found
-        word_size  = word_size  or context.word_size
-        endianness = endianness or context.endianness
-        sign       = sign       or context.sign
+        word_size  = 'all' if word_size == 'all' else context.word_size
+        endianness = context.endianness
+        sign       = context.sign
 
         if not isinstance(number, (int,long)):
             raise ValueError("pack(): number must be of type (int,long) (got %r)" % type(number))


Test patch:
Repository: Gallopsled/pwntools

Issue:
Doctests appear in `shellcraft foo.bar -?` output
This makes it hard to read the doctests.

```
$ shellcraft i386.linux.syscall -?
Args: [syscall_number, *args]
    Does a syscall

Example:

SPAM SPAM SPAM
```



Discussion:

Resolving patch:
diff --git a/pwnlib/commandline/shellcraft.py b/pwnlib/commandline/shellcraft.py
--- a/pwnlib/commandline/shellcraft.py
+++ b/pwnlib/commandline/shellcraft.py
@@ -164,7 +164,41 @@ def main():
         func = vals[0][1]
 
     if args.show:
-        print func.__doc__
+        # remove doctests
+        doc = []
+        in_doctest = False
+        block_indent = None
+        caption = None
+        for i, line in enumerate(func.__doc__.splitlines()):
+            if line.lstrip().startswith('>>>'):
+                # this line starts a doctest
+                in_doctest = True
+                if caption:
+                    # delete back up to the caption
+                    doc = doc[:caption - i]
+                    caption = None
+            elif line == '':
+                # skip blank lines
+                pass
+            elif in_doctest:
+                # indentation marks the end of a doctest
+                indent = len(line) - len(line.lstrip())
+                if block_indent is None:
+                    block_indent = indent
+                elif indent < block_indent:
+                    in_doctest = False
+                    block_indent = None
+            elif line.endswith(':'):
+                # save index of caption
+                caption = i
+            else:
+                # this is not blank space and we're not in a doctest, so the
+                # previous caption (if any) was not for a doctest
+                caption = None
+
+            if not in_doctest:
+                doc.append(line)
+        print '\n'.join(doc).rstrip()
         exit()
 
     defargs = len(func.func_defaults or ())


Test patch:
Repository: Gallopsled/pwntools

Issue:
bash completion for shellcraft would be nice
We used to have it.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -43,7 +43,13 @@
     name                 = 'pwntools',
     packages             = find_packages(),
     version              = version,
-    data_files           = [('', ['LICENSE-pwntools.txt'])],
+    data_files           = [('',
+                             ['LICENSE-pwntools.txt',
+                             ]),
+                            ('/etc/bash_completion.d',
+                             ['extra/bash_completion.d/shellcraft',
+                             ])
+                            ],
     package_data         = {
         'pwnlib': [
             'data/crcsums.txt',


Test patch:
Repository: Gallopsled/pwntools

Issue:
A log handler should not be installed in lib-mode
According to the [logging labrary's documenation](https://docs.python.org/2/howto/logging.html#configuring-logging-for-a-library) a library should not install any log handlers (except [`NullHandler`](https://docs.python.org/2/library/logging.handlers.html#logging.NullHandler)).

I suggest we move installation of our log handler and setting the `'pwnlib'` logger's level to `pwn.__init__`.  That way we should also be able to get rid of a [circular reference and hack](https://github.com/Gallopsled/pwntools/blob/master/pwnlib/term/__init__.py#L74-L76).



Discussion:
I agree this is a good idea. Should I do it in #391?

Let's keep it separate, no?


Resolving patch:
diff --git a/pwn/__init__.py b/pwn/__init__.py
--- a/pwn/__init__.py
+++ b/pwn/__init__.py
@@ -106,6 +106,15 @@ def asbool(s):
     # put the terminal in rawmode unless NOTERM was specified
     if term_mode:
         term.init()
+    # install a log handler and turn logging all the way up
+    import pwnlib.log as log
+    import logging
+    log.rootlogger.setLevel(logging.DEBUG)
+    handler = log.Handler()
+    formatter = log.Formatter()
+    handler.setFormatter(formatter)
+    log.rootlogger.addHandler(handler)
+
 closure()
 del closure
 
diff --git a/pwnlib/log.py b/pwnlib/log.py
--- a/pwnlib/log.py
+++ b/pwnlib/log.py
@@ -558,7 +558,8 @@ def getLogger(name):
     return _loggers[name]
 
 #
-# By default, everything will log to the console.
+# The root 'pwnlib' logger is declared here.  To change the target of all
+# 'pwntools'-specific logging, only this logger needs to be changed.
 #
 # Logging cascades upward through the heirarchy,
 # so the only point that should ever need to be
@@ -568,23 +569,5 @@ def getLogger(name):
 #     map(rootlogger.removeHandler, rootlogger.handlers)
 #     logger.addHandler(myCoolPitchingHandler)
 #
-#
-_console = Handler()
-_console.setFormatter(Formatter())
 
-#
-# The root 'pwnlib' logger is declared here, and attached to the
-# console.  To change the target of all 'pwntools'-specific
-# logging, only this logger needs to be changed.
-#
-# As explained at the top of this file we monkey patch the logger such that its
-# log level will be `logging.DEBUG` or `logging.INFO` depending on
-# `context.log_level`, if not set explicitly.
-#
-# Since properties are looked up on an instance's class we need to monkey patch
-# the whole class...
-rootlogger = logging.getLogger('pwnlib')
-rootlogger.setLevel(logging.DEBUG)
-rootlogger.addHandler(_console)
-rootlogger = Logger(rootlogger)
-_loggers['pwnlib'] = rootlogger
+rootlogger = getLogger('pwnlib')
diff --git a/pwnlib/term/__init__.py b/pwnlib/term/__init__.py
--- a/pwnlib/term/__init__.py
+++ b/pwnlib/term/__init__.py
@@ -71,8 +71,4 @@ def update_geometry():
     term.on_winch.append(update_geometry)
     readline.init()
 
-    # the logging module has an old reference to sys.stderr, so lets update that
-    from ..log import _console
-    _console.stream = sys.stderr
-
     term_mode = True


Test patch:
Repository: Gallopsled/pwntools

Issue:
Running shellcraft creates an `a.out` in $PWD
```
~> shellcraft i386.nop
90
~> ls -la a.out 
-rw-r--r-- 1 br0ns br0ns 664 Jan 10 16:28 a.out
```



Discussion:

Resolving patch:
diff --git a/pwnlib/asm.py b/pwnlib/asm.py
--- a/pwnlib/asm.py
+++ b/pwnlib/asm.py
@@ -152,7 +152,7 @@ def _assembler():
 
     if not checked_assembler_version[gas]:
         checked_assembler_version[gas] = True
-        result = subprocess.check_output([gas, '-v','/dev/null'],
+        result = subprocess.check_output([gas, '--version','/dev/null'],
                                          stderr=subprocess.STDOUT)
         version = re.search(r' (\d\.\d+)', result).group(1)
         if version < '2.19':


Test patch:
Repository: Gallopsled/pwntools

Issue:
History search crashes if item isn't found and enter is pressed
The history search in term.readline crashes when an item is searched for, but not found and then enter is pressed.

```
Traceback (most recent call last):
  File "./t.py", line 58, in <module>
    code.interact(local=locals())
  File "/usr/lib/python2.7/code.py", line 306, in interact
    console.interact(banner)
  File "/usr/lib/python2.7/code.py", line 234, in interact
    line = self.raw_input(prompt)
  File "./t.py", line 48, in _raw_input_wrapper
    ret = raw_input(prompt)
  File "./pwntools/pwnlib/term/readline.py", line 426, in raw_input
    return readline(None, prompt, float)
  File "./pwntools/pwnlib/term/readline.py", line 374, in readline
    keymap.handle_input()
  File "./pwntools/pwnlib/term/keymap.py", line 19, in handle_input
    self.send(key.get())
  File "./pwntools/pwnlib/term/keymap.py", line 47, in send
    cb(self.trace)
  File "./pwntools/pwnlib/term/readline.py", line 228, in submit
    commit_search()
  File "./pwntools/pwnlib/term/readline.py", line 140, in commit_search
    set_buffer(history[search_results[search_idx][0]], u'')
IndexError: list index out of range
```

It appears to be related to search_results being empty, but search_idx is set to 0.



Discussion:

Resolving patch:
diff --git a/pwnlib/term/readline.py b/pwnlib/term/readline.py
--- a/pwnlib/term/readline.py
+++ b/pwnlib/term/readline.py
@@ -136,7 +136,7 @@ def cancel_search(*_):
 
 def commit_search():
     global search_idx
-    if search_idx is not None:
+    if search_idx is not None and search_results:
         set_buffer(history[search_results[search_idx][0]], u'')
         search_idx = None
         redisplay()


Test patch:
Repository: Gallopsled/pwntools

Issue:
vma in asm function does not work.
asm("jmp $", vma=0x41410000)

it seems like its trying to make a 0x41410000 bytes big file.



Discussion:
I am not sure the gnu assembler actually supports this feature.

So this means that if we still wants to support this feature we need to link it to that address, right?


Resolving patch:
diff --git a/pwnlib/asm.py b/pwnlib/asm.py
--- a/pwnlib/asm.py
+++ b/pwnlib/asm.py
@@ -163,6 +163,21 @@ def _assembler():
 
     return assembler
 
+def _linker():
+    ld  = [which_binutils('ld')]
+    bfd = ['--oformat=' + _bfdname()]
+
+    E = {
+        'big':    '-EB',
+        'little': '-EL'
+    }[context.endianness]
+
+    arguments = {
+        'i386': ['-m', 'elf_i386'],
+    }.get(context.arch, [])
+
+    return ld + bfd + [E] + arguments
+
 def _objcopy():
     return [which_binutils('objcopy')]
 
@@ -349,8 +364,9 @@ def asm(shellcode, vma = 0, **kwargs):
 
     with context.local(**kwargs):
         assembler = _assembler()
+        linker    = _linker()
         objcopy   = _objcopy() + ['-j', '.shellcode', '-Obinary']
-        code      = '.org %#x\n' % vma
+        code      = ''
         code      += _arch_header()
         code      += cpp(shellcode)
 
@@ -360,6 +376,7 @@ def asm(shellcode, vma = 0, **kwargs):
         step1     = path.join(tmpdir, 'step1')
         step2     = path.join(tmpdir, 'step2')
         step3     = path.join(tmpdir, 'step3')
+        step4     = path.join(tmpdir, 'step4')
 
         try:
             with open(step1, 'w') as fd:
@@ -367,17 +384,27 @@ def asm(shellcode, vma = 0, **kwargs):
 
             _run(assembler + ['-o', step2, step1])
 
-            if file(step2,'rb').read(4) == '\x7fELF':
+            if not vma:
+                shutil.copy(step2, step3)
+
+            if vma:
+                 _run(linker + ['--section-start=.shellcode=%#x' % vma,
+                                '--entry=%#x' % vma,
+                                '-o', step3, step2])
+
+            elif file(step2,'rb').read(4) == '\x7fELF':
                 # Sanity check for seeing if the output has relocations
                 relocs = subprocess.check_output(
                     [which_binutils('readelf'), '-r', step2]
                 ).strip()
                 if len(relocs.split('\n')) > 1:
                     log.error('Shellcode contains relocations:\n%s' % relocs)
+            else:
+                shutil.copy(step2, step3)
 
-            _run(objcopy + [step2, step3])
+            _run(objcopy + [step3, step4])
 
-            with open(step3) as fd:
+            with open(step4) as fd:
                 result = fd.read()
 
         except:


Test patch: