internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Gallopsled/pwntools

Issue:
Bpython
Hi,

Unfortunately pwntools doesn't seem to work with bpython 0.12 in conjunction of python 2.7.9.

from pwn import *

results in:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pwn/**init**.py", line 2, in <module>
    from .toplevel import *
  File "/usr/local/lib/python2.7/dist-packages/pwn/toplevel.py", line 2, in <module>
    from pwnlib                  import *
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/**init**.py", line 10, in <module>
    from . import \
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/asm.py", line 45, in <module>
    from . import log
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/log.py", line 69, in <module>
    from .term      import spinners, text
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/term/**init**.py", line 1, in <module>
    from . import key, readline, text, termcap, keymap, term
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/term/readline.py", line 2, in <module>
    from . import term, text
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/term/text.py", line 111, in <module>
    sys.modules[**name**] = Module()
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/term/text.py", line 22, in **init**
    self.num_colors = termcap.get('colors', default = 8)
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/term/termcap.py", line 15, in get
    init()
  File "/usr/local/lib/python2.7/dist-packages/pwnlib/term/termcap.py", line 39, in init
    curses.setupterm()
TypeError: argument must be an int, or have a fileno() method.



Discussion:

Resolving patch:
diff --git a/pwnlib/term/termcap.py b/pwnlib/term/termcap.py
--- a/pwnlib/term/termcap.py
+++ b/pwnlib/term/termcap.py
@@ -36,6 +36,10 @@ def init():
     global cache
 
     if 'PWNLIB_NOTERM' not in os.environ:
-        curses.setupterm()
+        # Fix for BPython
+        try:
+            curses.setupterm()
+        except:
+            pass
 
     cache = {}


Test patch:
Repository: Gallopsled/pwntools

Issue:
`gdb.attach` does not remove temporary files
My `/tmp` is filled with `pwnXXXXXX.gdb` -files.



Discussion:
I don't think there's a nice way to guarantee these get deleted, but I agree we should probably remove them.  My vote is to delete them only once the child process spawned by `run_in_new_terminal` exits.

Sometimes things are broken and I need to inspect those files.

There's not a way to do this from GDB (i.e. an on-gdb-exit handler), but we should just be able to install a SIGCHLD handler.

Why is it a simple `atexit.register(os.unlink, tmp.name)` won't work again?


Resolving patch:
diff --git a/pwnlib/gdb.py b/pwnlib/gdb.py
--- a/pwnlib/gdb.py
+++ b/pwnlib/gdb.py
@@ -4,6 +4,7 @@
 import shlex
 import tempfile
 
+from . import atexit
 from . import elf
 from . import tubes
 from .asm import make_elf
@@ -32,6 +33,7 @@ def debug_shellcode(data, execute=None, **kwargs):
             f.write(elf_data)
             f.flush()
         os.chmod(tmp_elf, 0777)
+        atexit.register(lambda: os.unlink(tmp_elf))
         return debug(tmp_elf, execute=None, arch=context.arch)
 
 def get_qemu_arch(arch):
@@ -304,6 +306,7 @@ def findexe():
                                           delete = False)
         tmp.write(execute)
         tmp.close()
+        atexit.register(lambda: os.unlink(tmp.name))
         cmd += ' -x "%s" ; rm "%s"' % (tmp.name, tmp.name)
 
     log.info('running in new terminal: %s' % cmd)


Test patch:
Repository: Gallopsled/pwntools

Issue:
A little bug in Buffer class
There is a litttle bug in pwnlib.tubes.Buffer class.The class method unget has a type error in  line 117.add a buffer and a list

```
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    a.unget(b)
  File "buffer.py", line 117, in unget
    self.data = data + self.data
TypeError: unsupported operand type(s) for +: 'Buffer' and 'list'
```



Discussion:

Resolving patch:
diff --git a/pwnlib/tubes/buffer.py b/pwnlib/tubes/buffer.py
--- a/pwnlib/tubes/buffer.py
+++ b/pwnlib/tubes/buffer.py
@@ -114,7 +114,7 @@ def unget(self, data):
             'goodbyeworld'
         """
         if isinstance(data, Buffer):
-            self.data = data + self.data
+            self.data = data.data + self.data
             self.size += data.size
         else:
             self.data.insert(0, data)


Test patch:
Repository: Gallopsled/pwntools

Issue:
Don't fail on unknown (numeric) ELF segment types
Some executables can have unknown/non-standard ELF segment types (like `PT_PAX_FLAGS`) which show up as a numeric `p_type`, this makes the ELF class fail upon loading one of those files.

This uses an exact match (on `'PT_GNU_STACK'` and `'PT_GNU_RELRO'`) rather than a substring match so the comparison doesn't fail on integers.



Discussion:
@zachriggle: Is this still relevant after #636?


Resolving patch:
diff --git a/pwnlib/elf/elf.py b/pwnlib/elf/elf.py
--- a/pwnlib/elf/elf.py
+++ b/pwnlib/elf/elf.py
@@ -736,7 +736,7 @@ def nx(self):
 
         # Can't call self.executable_segments because of dependency loop.
         exec_seg = [s for s in self.segments if s.header.p_flags & P_FLAGS.PF_X]
-        return not any('GNU_STACK' in seg.header.p_type for seg in exec_seg)
+        return not any('GNU_STACK' in str(seg.header.p_type) for seg in exec_seg)
 
     @property
     def execstack(self):


Test patch:
Repository: Gallopsled/pwntools

Issue:
Need import
util/iters.py have not import time and context,It cause a problem when use mbruteforce

Need import
util/iters.py have not import time and context,It cause a problem when use mbruteforce

Need import
util/iters.py have not import time and context,It cause a problem when use mbruteforce



Discussion:

Resolving patch:
diff --git a/pwnlib/util/iters.py b/pwnlib/util/iters.py
--- a/pwnlib/util/iters.py
+++ b/pwnlib/util/iters.py
@@ -58,8 +58,10 @@
 import multiprocessing
 import operator
 import random
+import time
 from itertools import *
 
+from ..context import context
 from ..log import getLogger
 
 log = getLogger(__name__)


Test patch:
Repository: Gallopsled/pwntools

Issue:
"sh" shellcraft does not pass argv0
EOM;



Discussion:
It actually works fine.  It doesn't work with Busybox (Toybox for Android) since those both fail when `argv[0]` is NULL.

``` py
>>> context.arch='amd64'
>>> print shellcraft.sh()
    /* push '/bin///sh\x00' */
    push 0x68
    mov rax, 0x732f2f2f6e69622f
    push rax

    /* call execve('rsp', 0, 0) */
    mov rdi, rsp
    xor esi, esi
    push 0x3b
    pop rax
    cdq /* Set rdx to 0, rax is known to be positive */
    syscall

>>> write('sh.asm', shellcraft.sh())
```

This code is valid, and can be tested.  If you build it with Binjitsu (just used since it easy easy to turn it into an ELF)...

``` sh
$ asm -i sh.asm -c amd64 -f elf > sh.pwntools
```

...you can see that it works correctly.

``` sh
$ strace -e execve ./sh.pwntools
execve("./sh.pwntools", ["./sh.pwntools"], [/* 90 vars */]) = 0
execve("/bin///sh", [0], [/* 0 vars */]) = 0
```

However, Busybox requires `argv0` to be set, or else it doesn't know what tool to run.

I vote that we use a slightly longer shellcode that works even when the system uses busybox.

What do you think, @zachriggle? We already have that shellcode in android, right?

Correct, the code is slightly different for Linux and Android, due to busybox/toybox.

```
    ${arm.linux.execve('/bin///sh', 0, 0)}
```

vs.

```
    ${arm.linux.execve('/system/bin//sh', ['sh'], 0)}
```

The emitted code is _MUCH_ larger.

```
$ shellcraft i386.linux.sh
6a68682f2f2f73682f62696e6a0b5889e331c999cd80
```

```
$ shellcraft i386.android.sh
68010101018134247269010131d2526a045a01e25289e268010101018134242e7269016862696e2f6874656d2f682f7379736a0b5889e389d199cd80
```

Could we have an argument for this?

This is already currently possible with the `execve` shellcode.

```
$ shellcraft i386.linux.execve "/bin/sh" "['argv0', 'argv1']" "{'env':'value'}"
```

I'm not opposed to having a `sh` argument if it really helps

I think having an explicit argument would be nice in a stressed CTF situation. It does not have to be done by 3.0.


Resolving patch:
diff --git a/pwnlib/asm.py b/pwnlib/asm.py
--- a/pwnlib/asm.py
+++ b/pwnlib/asm.py
@@ -27,8 +27,8 @@
     Finally, :func:`asm` is used to assemble shellcode provided by ``pwntools``
     in the :mod:`shellcraft` module.
 
-        >>> asm(shellcraft.sh())
-        'jhh///sh/binj\x0bX\x89\xe31\xc9\x99\xcd\x80'
+        >>> asm(shellcraft.nop())
+        '\x90'
 
 Disassembly
 ------------------------


Test patch:
Repository: Gallopsled/pwntools

Issue:
3.0.3 Release Broken
It appears that the archive uploaded to PyPI does not include README.md, which is referred to by setup.py.

@Idolf can you update the release to include the README?



Discussion:
I've pulled the 3.0.3 release from PyPi.  I also deleted the tag, since auto-update would suggest that instead.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -77,8 +77,7 @@
     packages             = find_packages(),
     version              = '3.0.3',
     data_files           = [('',
-                             ['LICENSE-pwntools.txt',
-                             ]),
+                             glob.glob('*.md') + glob.glob('*.txt')),
                             ],
     package_data         = {
         'pwnlib': [


Test patch:
Repository: Gallopsled/pwntools

Issue:
Private functions and methods (underscore-prefixed) are not tested
Apparently the Sphinx `doctest` module does not actually run all of the tests:

```py
    def filter_members(self, members, want_all):
        # type: (List[Tuple[unicode, Any]], bool) -> List[Tuple[unicode, Any, bool]]
        """Filter the given member list.
        Members are skipped if
        - they are private (except if given explicitly or the private-members
          option is set)
        - they are special methods (except if given explicitly or the
          special-members option is set)
        - they are undocumented (except if the undoc-members option is set)
        The user can override the skipping decision by connecting to the
        ``autodoc-skip-member`` event.
        """
```

https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/autodoc.py#L855-L861


Discussion:

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -358,4 +358,9 @@ def linkcode_resolve(domain, info):
 # -- Customization to Sphinx autodoc generation --------------------------------------------
 import sphinx.ext.autodoc
 
-sphinx.ext.autodoc
+def dont_skip_any_doctests(app, what, name, obj, skip, options):
+    return False
+
+if 'doctest' in sys.argv:
+    def setup(app):
+        app.connect('autodoc-skip-member', dont_skip_any_doctests)
diff --git a/pwnlib/constants/__init__.py b/pwnlib/constants/__init__.py
--- a/pwnlib/constants/__init__.py
+++ b/pwnlib/constants/__init__.py
@@ -39,6 +39,16 @@
     ...     print int(constants.SYS_stat)
     4
 
+    >>> with context.local(arch = 'i386', os = 'linux'):
+    ...    print constants.SYS_execve + constants.PROT_WRITE
+    13
+    >>> with context.local(arch = 'amd64', os = 'linux'):
+    ...    print constants.SYS_execve + constants.PROT_WRITE
+    61
+    >>> with context.local(arch = 'amd64', os = 'linux'):
+    ...    print constants.SYS_execve + constants.PROT_WRITE
+    61
+
 """
 from __future__ import absolute_import
 
@@ -56,19 +66,6 @@ class ConstantsModule(ModuleType):
     ModuleType specialization in order to automatically
     route queries down to the correct module based on the
     current context arch / os.
-
-        >>> with context.local(arch = 'i386', os = 'linux'):
-        ...    print constants.SYS_execve + constants.PROT_WRITE
-        13
-        >>> with context.local(arch = 'amd64', os = 'linux'):
-        ...    print constants.SYS_execve + constants.PROT_WRITE
-        61
-        >>> with context.local(arch = 'amd64', os = 'linux'):
-        ...    print constants.SYS_execve + constants.PROT_WRITE
-        61
-        >>> False
-        True
-
     """
     Constant = Constant
 


Test patch:
Repository: Gallopsled/pwntools

Issue:
pwnlib.util.misc.run_in_terminal does not respect $TERM_PROGRAM
The documentation for `pwnlib.util.misc.run_in_terminal` asserts that it will respect the environment variable $TERM_PROGRAM, however that does not actually occur.

There's no reference to this environment variable when detecting which terminal to use:

```    if not terminal:
        if context.terminal:
            terminal = context.terminal[0]
            args     = context.terminal[1:]
        elif which('pwntools-terminal'):
            terminal = 'pwntools-terminal'
            args     = []
        elif 'DISPLAY' in os.environ and which('x-terminal-emulator'):
            terminal = 'x-terminal-emulator'
            args     = ['-e']
        elif 'TMUX' in os.environ and which('tmux'):
            terminal = 'tmux'
            args     = ['splitw']```


Discussion:
Huh! Whoops. Will fix soon, but not before the end of DEFCON!
On Sat, Apr 29, 2017 at 11:44 AM David Tomaschik <notifications@github.com>
wrote:

> The documentation for pwnlib.util.misc.run_in_terminal asserts that it
> will respect the environment variable $TERM_PROGRAM, however that does not
> actually occur.
>
> There's no reference to this environment variable when detecting which
> terminal to use:
>
>         if context.terminal:
>             terminal = context.terminal[0]
>             args     = context.terminal[1:]
>         elif which('pwntools-terminal'):
>             terminal = 'pwntools-terminal'
>             args     = []
>         elif 'DISPLAY' in os.environ and which('x-terminal-emulator'):
>             terminal = 'x-terminal-emulator'
>             args     = ['-e']
>         elif 'TMUX' in os.environ and which('tmux'):
>             terminal = 'tmux'
>             args     = ['splitw']```
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/Gallopsled/pwntools/issues/964>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAG0GOYoZQw1Vn1l7c5VYiC0wE7scguTks5r02kCgaJpZM4NMT5J>
> .
>


Resolving patch:
diff --git a/pwnlib/util/misc.py b/pwnlib/util/misc.py
--- a/pwnlib/util/misc.py
+++ b/pwnlib/util/misc.py
@@ -210,6 +210,9 @@ def run_in_new_terminal(command, terminal = None, args = None):
         elif which('pwntools-terminal'):
             terminal = 'pwntools-terminal'
             args     = []
+        elif 'TERM_PROGRAM' in os.environ:
+            terminal = os.environ['TERM_PROGRAM']
+            args     = []
         elif 'DISPLAY' in os.environ:
             terminal = 'x-terminal-emulator'
             args     = ['-e']


Test patch:
Repository: Gallopsled/pwntools

Issue:
pwn.conf skips validation checks
This blows up when the target architecture is 32-bit (e.g. `i386`) since we assemble a 64-bit object file and attempt to link it to a 32-bit ELF.

This is a somewhat expected result, since i386-64 doesn't exist.

```
/usr/local/bin/i386-unknown-linux-gnu-ld: i386:x86-64 architecture of input file `/var/folders/4h/rrwj8fpj1cqcfb83syr_f930008rd2/T/pwn-asm-eSdmd1/step2-obj' is incompatible with i386 output
```

While this is a user error, we should probably warn about this scenario.  It likely applies in lots of situations -- I think the best place to handle this is in the `LocalContext` decorator.  This ensures that we don't explode immediately on `context.bits` assignment (since it is valid to set `bits` before `arch`), but the user gets warned.


Discussion:
Apparently this is a result of `.pwn.conf` not triggering all of the validation routines, since we just replace the defaults.

Resolving patch:
diff --git a/pwnlib/context/__init__.py b/pwnlib/context/__init__.py
--- a/pwnlib/context/__init__.py
+++ b/pwnlib/context/__init__.py
@@ -1359,11 +1359,20 @@ def LocalContext(function):
     """
     @functools.wraps(function)
     def setter(*a, **kw):
-        # Fast path to skip adding a Context frame
-        if not kw:
-            return function(*a)
-
         with context.local(**{k:kw.pop(k) for k,v in kw.items() if isinstance(getattr(ContextType, k, None), property)}):
+            arch = context.arch
+            bits = context.bits
+            end = context.endian
+
+            # Prevent the user from doing silly things with invalid
+            # architecture / bits / endianness combinations.
+            if (arch == 'i386' and bits != 32) \
+              or (arch == 'amd64' and bits != 64):
+                raise AttributeError("Invalid arch/bits combination: %s/%s" % (arch, bits))
+
+            if arch in ('i386', 'amd64') and endianness == 'big':
+                raise AttributeError("Invalid arch/endianness combination: %s/%s" % (arch, endian))
+
             return function(*a, **kw)
     return setter
 
@@ -1385,6 +1394,15 @@ def update_context_defaults(section):
         else:
             log.warn("Unsupported configuration option %r in section %r" % (key, 'context'))
 
-        ContextType.defaults[key] = type(default)(value)
+        # Attempt to set the value, to see if it is value:
+        try:
+            value = type(default)(value)
+            with context.local(**{key: value}):
+                pass
+        except (ValueError, AttributeError) as e:
+            log.warn("Could not set context.%s=%s via pwn.conf (%s)", key, section[key], e)
+            continue
+
+        ContextType.defaults[key] = value
 
 register_config('context', update_context_defaults)


Test patch:
Repository: Gallopsled/pwntools

Issue:
DynELF._find_linkmap bug
https://github.com/Gallopsled/pwntools/blob/b53e58a840c890340fa755a2229c3ff774c5f81e/pwnlib/dynelf.py#L450

Bug after commit bb25140807a4d
Must be:

```python
w.status("r_debug.linkmap %#x" % linkmap)
```


Discussion:

Resolving patch:
diff --git a/pwnlib/dynelf.py b/pwnlib/dynelf.py
--- a/pwnlib/dynelf.py
+++ b/pwnlib/dynelf.py
@@ -447,7 +447,7 @@ def _find_linkmap(self, pltgot=None, debug=None):
             if debug:
                 w.status("r_debug.linkmap")
                 linkmap = self.leak.field(debug, r_debug.r_map)
-                w.status("r_debug.linkmap %#x" % result)
+                w.status("r_debug.linkmap %#x" % linkmap)
 
         if not linkmap:
             w.failure("Could not find DT_PLTGOT or DT_DEBUG")


Test patch:
Repository: Gallopsled/pwntools

Issue:
ssh_process.elf can return incorrect ELF, when ssh_process.executable is included in the ssh cache path
When getting the target process' ELF using the `ssh_process.elf` property, a wrong ELF is returned, when the path of the target binary on the remote system is included in the path of the local cache:

Example:
`target.libs()` returns:

```python
{'/somepath/target/target1/somehost.org/lib32/libc.so.6': 4158791680, 
 '/somepath/target/target1/somehost.org/target/target1': 0, 
 '/somepath/target/target1/somehost.org/lib/ld-linux.so.2': 4160602112}
```

while `target.executable` is `/target/target1`.
So `/target/target1` is included in the cache path.

Therefore the condition evaluates to true (just checks whether the string is contained): https://github.com/Gallopsled/pwntools/blob/df7c1fef9d51b4f5fe4fe13c3394168b30e9d7a4/pwnlib/tubes/ssh.py#L343-L345

Maybe the condition should check 
```py
lib.endswith(self.executable)
```
instead. However I am not 100% sure, whether this covers all cases.


Discussion:
Good catch!  

I'm not sure how to resolve this perfectly, since it's also possible for a binary to rely on another ELF with the same `basename`.  In the meantime, using `basename` is probably as good as we can do -- would you mind submitting a pull request? ❤️ 

Thanks!

Resolving patch:
diff --git a/pwnlib/tubes/ssh.py b/pwnlib/tubes/ssh.py
--- a/pwnlib/tubes/ssh.py
+++ b/pwnlib/tubes/ssh.py
@@ -341,7 +341,8 @@ def elf(self):
         libs = self.parent.libs(self.executable)
 
         for lib in libs:
-            if self.executable in lib:
+            # Cannot just check "executable in lib", see issue #1047
+            if lib.endswith(self.executable):
                 return pwnlib.elf.elf.ELF(lib)
 
 


Test patch:
Repository: Gallopsled/pwntools

Issue:
Travis tests are broken
Looks like tests broke somehow.  

We didn't change anything that would cause this, something in the dependency / testing stack changed.

Good: https://travis-ci.org/Gallopsled/pwntools/builds/338641937
Bad: https://travis-ci.org/Gallopsled/pwntools/builds/341197032

Error message:
```
$ PWNLIB_NOTERM=1 coverage run -m sphinx -b doctest docs/source docs/build/doctest
Running Sphinx v1.7.0
Extension error:
Could not import extension pwnlib.internal.dochelper (exception: No module named compat)
Coverage.py warning: Module ~/.pwntools-cache/ was never imported. (module-not-imported)
The command "PWNLIB_NOTERM=1 coverage run -m sphinx -b doctest docs/source docs/build/doctest" exited with 2.
```

Separately, we should probably have something that auto-files a bug when tests break.



Discussion:
@Idolf this will likely block the next release

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -24,6 +24,27 @@
 
 import pwnlib
 
+# -- WORK-AROUNDS FOR DEPRECATION ----------------------------------------------
+# Deprecated
+# 1.6b1
+# sphinx.util.compat.Directive class is now deprecated.
+# Please use instead docutils.parsers.rst.Directive
+#
+# Pwntools Note:
+#   Can't just "do the right thing" since we have dependencies that
+#   are also affected by this, specifically sphinxcontrib.autoprogram
+try:
+    import sphinx.util.compat
+except ImportError:
+    import sys
+    import types
+    import sphinx.util
+    import docutils.parsers.rst
+    class compat(types.ModuleType):
+        Directive = docutils.parsers.rst.Directive
+    sphinx.util.compat = compat('sphinx.util.compat')
+    sys.modules['sphinx.util.compat'] = sphinx.util.compat
+
 # -- General configuration -----------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.


Test patch: