Repository: secdev/scapy

Issue:
[Windows] POWERSHELL_MANAGER is never closed explicitly
scapy-2.4.0rc4.dev58-py2.7
Windows 7

POWERSHELL_MANAGER isn't closed in arch/windows/__init__.py

That's a problem, because subprocess used inherits open files. The files can't be deleted without waiting for scapy to end (sleep-wait-repeat).

I've added POWERSHELL_MANAGER.close() near the bottom of __init__.py and it solved the problem.

Steps to reproduce:
1. Create a file.
2. Replace sys.stdout/stderr with it (I'm doing a tee)
3. Import scapy
4. Try to remove created file after closing it

Observed behaviour:
- exception - file in use, can't be deleted

Expected behaviour:
- file is deleted just fine

Reproducible:
- always


Discussion:
@rgom Hi ! Thanks a lot for reporting

> Try to remove created file after closing it

Do you mean closing the file or scapy ?

Thanks !
Hello,
I mean closing the file.

If I create a file:

a = open('...', 'w')

then do a subprocess:

subprocess.Popen()...

then close the file:

a.close()

the file can't be deleted with:

os.remove(a.name)

Please compare to Python bug report https://bugs.python.org/issue19575
This is identical story, but all the enhancements went to Python 3.

I can see no other way to proceed than proposed (add the close explicit call).
If anybody can come up with anything smarter, I'd appreciate.

HTH,
Robert
```import os
f = open('c:/temp/x.txt', 'w')
from scapy import all as scapy  # TROUBLE AHEAD
f.close()
os.remove(f.name)
```

```
Traceback (most recent call last):
  File "D:\A.py", line 9, in <module>
    os.remove(f.name)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:/temp/x.txt'
```
I understand what you mean.

The point of not calling `POWERSHELL_PROCESS.close()` is that scapy does not have to start any more shell while performing OS access (through cmd/powershell). It speeds up the process to have a shell in no-CPU mode in the background.

I'm looking into a solution :)
This is a Python 2.7 bug.
It was fixed on python 3.4+ ([link](https://www.python.org/dev/peps/pep-0433/)).

Scapy is not responsible for it... I'll try to find a work-around
Thank you for the feedback.

I must admit I have a different opinion, since I didn't observe that on any other imported package.

I have a workaround for my applications (adding NOINHERIT flag when opening a file created before importing scapy), but it's hard for me to maintain and I'd expect similar bug reports in the future.

I couldn't find anything relevant in subprocess myself (close_fds has significant limitations).

Regards,
Robert
It is good know issue/error in Py2.
There is no other option than setting this flag by ctypes for windows.
In Python 2.7 there is no sys.get/set_handle_inheritable() but is easy to write those.
Another way/workaround (on our side) is to wait for finishing thread (and - powershell - process in it).
On scapy side will be best of existing e.g. scapy.finalize() method which will be responsible for closing
appropriate resouces (e.g. POWERSHELL_PROCESS) and can be used in our code _before_ closing/renaming blocked file/handle.
Another workaround is to use scapy.???.???.POWERSHELL_PROCESS.close() in our code (in place when finalize() should be used) but it is weird, not portable etc. etc.
Simply: such way cannot be used on our side :(
Waiting for release handle is much better/natural (simply emulates waiting for semaphore.. Yes, it 
is de facto the same problem:).
Another way is use io.open() instead of open() but I don't know whether io.open() has the same
feature (not inherit handles in child processes) in PY2 as both (open() & io.open()) in PY3.
Anyway: worth to  check this out... but can be another problem with encoding etc...
io.open() & open(0 is not quite the same (especially in Py2).

UPDATE: unfortunately io.open() behaves the same way as open() on PY2 :(
Anyway:... 
Maybe worth to replace powershell/vbscript calls to an appropriate pure Python solutions ?
In one/two places is easy (e.g. reading Windows registry) in other will be harder but.. not impossible..
Is there a chance for doing it in short period for release 2.4? I doubt :(
I’m thinking of a hack to set all opened files as not herited. Should be possible with windll api calls.. will check that out tonight.

> Le 7 févr. 2018 à 11:41, Adam Karpierz <notifications@github.com> a écrit :
> 
> Anyway:...
> Maybe worth to replace powershell/vbscript callls for appropriate pure Python solutions ?
> In one/two places is easy (e.g. reading Windows registry) in other will be harder but.. not impossible..
> Is there a chance for doing it in short period for release 2.4? I doubt :(
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 

Maybe our hacks will be useful for you...

```
# ctypes's backports from PY3


def get_inheritable(fd):

    from ctypes import byref
    from ctypes import windll, wintypes
    from ctypes import WinError
    from msvcrt import get_osfhandle

    HANDLE_FLAG_INHERIT = 1

    handle = wintypes.HANDLE(get_osfhandle(fd))
    flags  = wintypes.DWORD()
    if not windll.kernel32.GetHandleInformation(handle, byref(flags)):
        raise WinError()

    return bool(flags.value & HANDLE_FLAG_INHERIT)


def set_inheritable(fd, inheritable):

    from ctypes import windll, wintypes
    from ctypes import WinError
    from msvcrt import get_osfhandle

    HANDLE_FLAG_INHERIT = 1

    handle = wintypes.HANDLE(get_osfhandle(fd))
    mask   = wintypes.DWORD(HANDLE_FLAG_INHERIT)
    flags  = wintypes.DWORD(HANDLE_FLAG_INHERIT if inheritable else 0)
    if not windll.kernel32.SetHandleInformation(handle, mask, flags):
        raise WinError()


with open("AAA.txt", "w") as f:
    print(get_inheritable(f.fileno()))
    set_inheritable(f.fileno(), False)
    print(get_inheritable(f.fileno()))


import os
with os.fdopen(os.open("XXX.txt", os.O_NOINHERIT | os.O_WRONLY | os.O_CREAT), "w") as f:
    print(get_inheritable(f.fileno()))
    from scapy import all as scapy  # TROUBLE AHEAD
os.remove("XXX.txt")

# Displays: False (means OK).
```

:: I’m thinking of a hack to set all opened files as not inherited.

subprocess.Popen(close_fds=True) will be helpful but then exist some problems with stdout/stderr redirection. 

Thanks,
Adam Karpierz
I got a working fix !
@karpierz @rgom 
Hi !
Issue was fixed in https://github.com/secdev/scapy/pull/1138
![image](https://user-images.githubusercontent.com/10530980/35926617-dd8a551a-0c28-11e8-91d1-8b9e5d98cfe2.png)

`close_fds` is not available when stdout/stderr is used. The provided hack converts already existing opened files to non inheritable ones.
@karpierz For the record, it's really hard for us not to call any CMD/VBS commands, without having to implement C code.

As scapy is a pure python project, it would become harder to compile, and break the `download-and-run` policy
I really can help in this area in the future because I am familiar with ctypes usage a years (and I am really very old C/C++ programmer;) 
Seriously: Practically all my low-level Python C-related code is based only and only on ctypes
so it is strict pure/standard Python.
PS:  In your place I would use my willingness, before I will gone to the land of eternal compilers ;)
@karpierz You are really welcomed to do so !
Thank for your fix a lot!! 
It was really blocked thing for us.
Thank you for the fix.
I confirm it fixes the test case provided - I'll have to verify it on bigger code base to see impact on actual code, but I don't expect surprises.
Is it going to land in 2.4 release?

Regards,
Robert
Update:
I've verified the change successfully on bigger code base.

Resolving patch:
diff --git a/scapy/arch/windows/__init__.py b/scapy/arch/windows/__init__.py
--- a/scapy/arch/windows/__init__.py
+++ b/scapy/arch/windows/__init__.py
@@ -12,7 +12,9 @@
 import os, re, sys, socket, time, itertools, platform
 import subprocess as sp
 from glob import glob
-import ctypes, tempfile
+import ctypes
+from ctypes import wintypes
+import tempfile
 from threading import Thread, Event
 
 import scapy
@@ -27,6 +29,18 @@
 from scapy.modules.six.moves import range, zip, input, winreg
 from scapy.compat import plain_str
 
+_winapi_SetConsoleTitle = ctypes.windll.kernel32.SetConsoleTitleW
+_winapi_SetConsoleTitle.restype = wintypes.BOOL
+_winapi_SetConsoleTitle.argtypes = [wintypes.LPWSTR]
+
+_winapi_GetHandleInformation = ctypes.windll.kernel32.GetHandleInformation
+_winapi_GetHandleInformation.restype = wintypes.BOOL
+_winapi_GetHandleInformation.argtypes = [wintypes.HANDLE, ctypes.POINTER(wintypes.DWORD)]
+
+_winapi_SetHandleInformation = ctypes.windll.kernel32.SetHandleInformation
+_winapi_SetHandleInformation.restype = wintypes.BOOL
+_winapi_SetHandleInformation.argtypes = [wintypes.HANDLE, wintypes.DWORD, wintypes.DWORD]
+
 conf.use_pcap = False
 conf.use_dnet = False
 conf.use_winpcapy = True
@@ -80,58 +94,79 @@ def _windows_title(title=None):
     """Updates the terminal title with the default one or with `title`
     if provided."""
     if conf.interactive:
-        title = title or "Scapy v" + conf.version
-        ctypes.windll.kernel32.SetConsoleTitleW(title)
+        _winapi_SetConsoleTitle(title or "Scapy v{}".format(conf.version))
 
-def _suppress_file_handles_inheritance(r=100):
+def _suppress_file_handles_inheritance(r=1000):
     """HACK: python 2.7 file descriptors.
 
     This magic hack fixes https://bugs.python.org/issue19575
+    and https://github.com/secdev/scapy/issues/1136
     by suppressing the HANDLE_FLAG_INHERIT flag to a range of
     already opened file descriptors.
+    Bug was fixed on python 3.4+
     """
-    # See https://github.com/secdev/scapy/issues/1136
+    if sys.version_info[0:2] >= (3, 4):
+        return []
+
     import stat
-    from ctypes import windll, wintypes
     from msvcrt import get_osfhandle
 
     HANDLE_FLAG_INHERIT = 0x00000001
 
+    handles = []
     for fd in range(r):
         try:
             s = os.fstat(fd)
         except OSError:
             continue
         if stat.S_ISREG(s.st_mode):
-            handle = wintypes.HANDLE(get_osfhandle(fd))
-            mask = wintypes.DWORD(HANDLE_FLAG_INHERIT)
-            flags = wintypes.DWORD(0)
-            windll.kernel32.SetHandleInformation(handle, mask, flags)
+            osf_handle = get_osfhandle(fd)
+            flags = wintypes.DWORD()
+            _winapi_GetHandleInformation(osf_handle, flags)
+            if flags.value & HANDLE_FLAG_INHERIT:
+                _winapi_SetHandleInformation(osf_handle, HANDLE_FLAG_INHERIT, 0)
+                handles.append(osf_handle)
+
+    return handles
+
+def _restore_file_handles_inheritance(handles):
+    """HACK: python 2.7 file descriptors.
+
+    This magic hack fixes https://bugs.python.org/issue19575
+    and https://github.com/secdev/scapy/issues/1136
+    by suppressing the HANDLE_FLAG_INHERIT flag to a range of
+    already opened file descriptors.
+    Bug was fixed on python 3.4+
+    """
+    if sys.version_info[0:2] >= (3, 4):
+        return
+
+    HANDLE_FLAG_INHERIT = 0x00000001
+
+    for osf_handle in handles:
+        try:
+            _winapi_SetHandleInformation(osf_handle, HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT)
+        except (ctypes.WinError, WindowsError, OSError):
+            pass
 
 class _PowershellManager(Thread):
     """Instance used to send multiple commands on the same Powershell process.
     Will be instantiated on loading and automatically stopped.
     """
     def __init__(self):
-        if sys.version_info[0:2] < (3, 4): # Bug was fixed on python 3.4+
-            # Fix https://bugs.python.org/issue19575
-            # and https://github.com/secdev/scapy/issues/1136
-            _suppress_file_handles_inheritance()
-        # Start & redirect input
-        if conf.prog.powershell:
-            self.process = sp.Popen([conf.prog.powershell,
-                                     "-NoLogo", "-NonInteractive",  # Do not print headers
-                                     "-Command", "-"],  # Listen commands from stdin
-                             stdout=sp.PIPE,
-                             stdin=sp.PIPE,
-                             stderr=sp.STDOUT)
-            self.cmd = False
-        else:  # Fallback on CMD (powershell-only commands will fail, but scapy use the VBS fallback)
-            self.process = sp.Popen([conf.prog.cmd],
-                             stdout=sp.PIPE,
-                             stdin=sp.PIPE,
-                             stderr=sp.STDOUT)
-            self.cmd = True
+        opened_handles = _suppress_file_handles_inheritance()
+        try:
+            # Start & redirect input
+            if conf.prog.powershell:
+                cmd = [conf.prog.powershell,
+                       "-NoLogo", "-NonInteractive",  # Do not print headers
+                       "-Command", "-"]  # Listen commands from stdin
+            else:  # Fallback on CMD (powershell-only commands will fail, but scapy use the VBS fallback)
+                cmd = [conf.prog.cmd]
+            self.process = sp.Popen(cmd, stdout=sp.PIPE, stdin=sp.PIPE, stderr=sp.STDOUT)
+            self.cmd = not conf.prog.powershell
+        finally:
+            _restore_file_handles_inheritance(opened_handles)
         self.buffer = []
         self.running = True
         self.query_complete = Event()


Test patch: