Repository: pyinstaller/pyinstaller

Issue:
Pyinstaller lock exe file while running on windows
Version of python: 2.7 32bit
Version of pyinstall: download from git

How to reproduce:
1: create 1.py, content:

```
import time
time.sleep(99999)
```

2: pack 1.py to 1.exe, run

```
pyinstaller.py 1.py
```

3: execute 1.exe

Then 1.exe cannot be rename while running.

I test py2exe and cx_Freeze they don't have this problem.
I think it's because pyinstaller didn't close file handle of exe after read it needs.

Pyinstaller use fopen not CreateFile so it has the default behavior.
other problem is pyinstaller will inhert file handles to child process but it's not necessarily,
I already test change py_utils.c:520 to FALSE is ok,
but problem still not resolved.

http://en.wikipedia.org/wiki/File_locking
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx

If you still have problem to read this please contact me instead of close, otherwise I will open other again.



Discussion:
@YueLunGames thanks for detailed report.

I was able to reproduced this behavior. How it should be fixed?
- just replace fopen calls by CreateFile?

Resolved.
Here is the patch, but I will not create the pull request because it's dirty fix.
It may decrease performance but will not hold the file handle for long.

Edit:
I said fopen has the default behavior I mean it will open file by dwShareMode = FILE_SHARE_READ | FILE_SHARE_WRITE, but no FILE_SHARE_DELETE.
I tried replace fopen to CreateFile but it's not help because pyi_archive.Archive also open the executable file for long, finally I use StringIO (which will cause memory usage increase mbs).

```
--- a/bootloader/common/pyi_archive.c.orig
+++ b/bootloader/common/pyi_archive.c
@@ -107,6 +107,15 @@ unsigned char *pyi_arch_extract(ARCHIVE_STATUS *status, TOC *ptoc)
    unsigned char *data;
    unsigned char *tmp;

+   /* Reopen file handler if need */
+   if (status->fp == NULL) {
+       status->fp = stb_fopen(status->archivename, "rb");
+       if (status->fp == NULL) {
+           VS("LOADER: Cannot open archive: %s\n", status->archivename);
+           return NULL;
+       }
+   }
+
    fseek(status->fp, status->pkgstart + ntohl(ptoc->pos), SEEK_SET);
    data = (unsigned char *)malloc(ntohl(ptoc->len));
    if (data == NULL) {
@@ -127,6 +136,10 @@ unsigned char *pyi_arch_extract(ARCHIVE_STATUS *status, TOC *ptoc)
            return NULL;
        }
    }
+   
+   /* close file handler to avoid locking */
+   fclose(status->fp);
+   status->fp = NULL;
    return data;
 }

@@ -353,6 +366,10 @@ int pyi_arch_open(ARCHIVE_STATUS *status)
        FATALERROR("Error on file");
        return -1;
    }
+   
+   /* Close file handle to avoid locking */
+   fclose(status->fp);
+   status->fp = NULL;
    return 0;
 }

@@ -433,6 +450,11 @@ void pyi_arch_status_free_memory(ARCHIVE_STATUS *archive_status)
         if (archive_status->tocbuff != NULL) {
             free(archive_status->tocbuff);
         }
+        /* Free file handler */
+        if (archive_status->fp != NULL) {
+            fclose(archive_status->fp);
+            archive_status->fp = NULL;
+        }
         free(archive_status);
     }
 }
--- a/PyInstaller/loader/pyi_archive.py.orig
+++ b/PyInstaller/loader/pyi_archive.py
@@ -88,7 +88,9 @@ class Archive(object):
         import imp
         self.pymagic = imp.get_magic()
         if path is not None:
-            self.lib = open(self.path, 'rb')
+            from cStringIO import StringIO
+            with open(self.path, 'rb') as fp:
+                self.lib = StringIO(fp.read())
             self.checkmagic()
             self.loadtoc()

@@ -121,7 +123,7 @@ class Archive(object):
         self.lib.seek(self.start + self.TOCPOS)
         (offset,) = struct.unpack('!i', self.lib.read(4))
         self.lib.seek(self.start + offset)
-        self.toc = marshal.load(self.lib)
+        self.toc = marshal.loads(self.lib.read())

     ######## This is what is called by FuncImporter #######
     ## Since an Archive is flat, we ignore parent and modname.
```

```
```

@YueLunGames Thanks for investigation.

I think, the performance impact is negligible.
I'm more concerned about reading the whole archive into memory and keeping it there for the whole runtime of the program.

In fact the file pointer should not be the member of a long living class.
File object lifetime should be managed as short as they can.

you can do 

```
with self.openfile() as lib:
    self.checkmagic(lib)
    self.loadtoc(lib)
```

and

```
with archive.openfile() as lib:
    archive.extract(lib, name)
```

@YueLunGames Could you please submit a pull-request with out changes. You can even fork on github, "edit" the files online (paste your changed file in) and submit the pull-request.Thanks.

ok, I will try to create an clean fix in next few days.

Thanks.

Since there is a related pull-request, I'm closing the issue.


Resolving patch:
diff --git a/PyInstaller/loader/pyimod02_archive.py b/PyInstaller/loader/pyimod02_archive.py
--- a/PyInstaller/loader/pyimod02_archive.py
+++ b/PyInstaller/loader/pyimod02_archive.py
@@ -28,6 +28,10 @@
 import struct
 import sys
 import zlib
+if sys.version_info[0] == 2:
+    import thread
+else:
+    import _thread as thread
 
 
 # For decrypting Python modules.
@@ -39,6 +43,17 @@
 PYZ_TYPE_PKG = 1
 PYZ_TYPE_DATA = 2
 
+class FilePos(object):
+    """
+    This class keeps track of the file object representing and current position
+    in a file.
+    """
+    def __init__(self):
+        # The file object representing this file.
+        self.file = None
+        # The position in the file when it was last closed.
+        self.pos = 0
+
 
 class ArchiveFile(object):
     """
@@ -49,41 +64,51 @@ class ArchiveFile(object):
     def __init__(self, *args, **kwargs):
         self.args = args
         self.kwargs = kwargs
-        self.pos = 0
-        self.fd = None
-        self.__open()
+        self._filePos = {}
 
-    def __getattr__(self, name):
+    def local(self):
         """
-        Auto open file when access member from file object
-        This function only call when member of name not exist in self
+        Return an instance of FilePos for the current thread. This is a crude
+        # re-implementation of threading.local, which isn't a built-in module
+        # and therefore isn't available.
         """
-        assert self.fd
-        return getattr(self.fd, name)
+        ti = thread.get_ident()
+        if ti not in self._filePos:
+            self._filePos[ti] = FilePos()
+        return self._filePos[ti]
 
-    def __open(self):
+    def __getattr__(self, name):
         """
-        Open file and seek to pos record from last close
+        Make this class act like a file, by invoking most methods on its
+        underlying file object.
         """
-        if self.fd is None:
-            self.fd = open(*self.args, **self.kwargs)
-            self.fd.seek(self.pos)
+        file = self.local().file
+        assert file
+        return getattr(file, name)
 
     def __enter__(self):
-        self.__open()
+        """
+        Open file and seek to pos record from last close.
+        """
+        # The file shouldn't be open yet.
+        fp = self.local()
+        assert not fp.file
+        # Open the file and seek to the last position.
+        fp.file = open(*self.args, **self.kwargs)
+        fp.file.seek(fp.pos)
 
     def __exit__(self, type, value, traceback):
-        assert self.fd
-        self.close()
-
-    def close(self):
         """
-        Close file and record pos
+        Close file and record pos.
         """
-        if self.fd is not None:
-            self.pos = self.fd.tell()
-            self.fd.close()
-            self.fd = None
+        # The file should still be open.
+        fp = self.local()
+        assert fp.file
+
+        # Close the file and record its position.
+        fp.pos = fp.file.tell()
+        fp.file.close()
+        fp.file = None
 
 
 class ArchiveReadError(RuntimeError):


Test patch:
diff --git a/tests/functional/test_libraries.py b/tests/functional/test_libraries.py
--- a/tests/functional/test_libraries.py
+++ b/tests/functional/test_libraries.py
@@ -250,7 +250,6 @@ def test_pygments(pyi_builder):
         print(highlight(code, PythonLexer(), HtmlFormatter()))
         """)
 
-
 @importorskip('markdown')
 def test_markdown(pyi_builder):
     # Markdown uses __import__ed extensions. Make sure these work by
diff --git a/tests/unit/test_hookutils.py b/tests/unit/test_hookutils.py
--- a/tests/unit/test_hookutils.py
+++ b/tests/unit/test_hookutils.py
@@ -11,12 +11,16 @@
 import os
 import pytest
 import shutil
-
 from os.path import join
+from threading import Thread
+from queue import Queue
+
 from PyInstaller.utils.hooks import collect_data_files, collect_submodules, \
   get_module_file_attribute, remove_prefix, remove_suffix, \
   remove_file_extension, is_module_or_submodule
 from PyInstaller.compat import exec_python
+from PyInstaller.loader.pyimod02_archive import ArchiveFile
+
 
 class TestRemovePrefix(object):
     # Verify that removing a prefix from an empty string is OK.
diff --git a/tests/unit/test_loader.py b/tests/unit/test_loader.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_loader.py
@@ -0,0 +1,60 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2005-2016, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License with exception
+# for distributing bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#-----------------------------------------------------------------------------
+
+
+import pytest
+from threading import Thread
+from queue import Queue
+
+from PyInstaller.loader.pyimod02_archive import ArchiveFile
+
+def test_threading_import(tmpdir):
+    """
+    On Python 3.3+, PyInstaller doesn't acquire a lock when performing an
+    import. Therefore, two thread could both be reading the .pyz archive at the
+    same time. At the core, the ArchiveFile class performs these reads. This
+    test verifies that multi-threaded reads work.
+
+    For more information, see https://github.com/pyinstaller/pyinstaller/pull/2010.
+    """
+
+    # Create a temporary file and use the ArchiveReader on it.
+    tmp_file = tmpdir.join('test.txt')
+    tmp_file.write('Testing')
+    ar = ArchiveFile(tmp_file.strpath, 'r')
+
+    # Use queues to synchronize threads.
+    q1 = Queue()
+    q2 = Queue()
+
+    # This function, which is run in a separate thread, works to ensure that
+    # both threads open a file at the same time.
+    def foo():
+        with ar:
+            # Wait until both threads have opened the file.
+            q1.put(1)
+            assert q2.get() == 2
+            # Wait until the other thread has closed the file before closing it
+            # here.
+            assert q2.get() == 3
+
+    thread = Thread(target=foo)
+    thread.start()
+
+    # This code works with ``foo`` above to open the same file from two threads.
+    with ar:
+        # Wait until both threads have opened the file.
+        q2.put(2)
+        assert q1.get() == 1
+    # Make the other thread wait until this thread has closed the file.
+    q2.put(3)
+
+    # Wait for the other thread to finish.
+    thread.join()
+