Repository: conan-io/conan

Issue:
Can an interrupted install leave locks inconsistent?
I have seen conan installs fail with a message like this:

```
<name>/<version>@<channel> is locked by another concurrent conan process, wait...

If not the case, quit, and do 'conan remove <name>/<version>@<channel> -f'
```

Trying to get to the root of this message I think I might have CTRL-C interrupted a previous installation, which in turn left the corresponding package lock file on disk. Then during another install attempt, the new process waited forever.

I looked into the conan code and did not find try/finally logic with respect to the lock context managers. Does that mean I can end up in an inconsistent situation like described? Wouldn't it be preferrable to clean up locks also in such a situation? Semantically this would be correct I think, since the lock does not mean "installed fine", but simply "no-one else is writing". In case of CTRL-C this is in fact true, the process stopped writing, so the lock should be gone as well.

Or did I get lost? :smiley: 


Discussion:
You are as lost as we all are regarding concurrency 😃😃 Well, this is an interesting issue, we have done a several attempts to improve.

Instead of using try-finally, we are using context managers, which in theory are also guaranteed to "__exit__" properly, and do their task. I have run a few tests with Ctrl+C, and in Windows it works.

Unfortunately, there seem to be some bad cases, for example SIGKILL in linux, which won't let the process to finish, so the locks will remain in some inconsistent state, no matter what conan tries to do.

So we (@SSE4 did the initial proposal, I collaborated) tried to use the process PIDs in the files, so we could know which processes were still alive and discard the others. It happens than the ``psutil`` library we were using has several limitations in different platforms, and also the PIDs approach is not totally valid, as some OSs as Windows reuse the PIDs from dead processes.

So so far, we are quite good with the message, the ``conan remove`` just do it. It would be good to learn in which system are you, and try to identify why the Ctrl+C signal might not be properly captured (to test I just prepare a package recipe with ``time.sleep(5)`` in each method.

Please tell me if this makes sense. Thanks!
Thanks a lot for the quick reply!

What you describe does make total sense, but then I should not see the issue. I am running the following setup:

* Windows 7 Enterprise 64 Bit, Windows 10 Pro 64 Bit (happened on both)
* CPython 3.6.3
* `conan install` is called as subprocess from within a [doit](http://pydoit.org/) task.

The workflow installs a number of dependencies on a developer machine or CI systems. For the CI I already made `CONAN_USER_HOME` job-local, but on my machine there should not be any concurrent conan calls.

I'll try to reproduce and post a recipe/procedure here. Thanks again!
Ok, looking forward your feedback then. Are you sure the doit is not parallelizing any conan calls? (even though, it should be ok, the main problem is killing processes)
Thanks!
While doit is able to execute processes in parallel, I am doing very few conan calls, strictly sequential:

1. `conan remote list`
2. `conan remote remove` (depending on list result)
3. `conan remote add` (depending on list result)
4. `conan user`
5. `conan install`

I have seen the error mainly popping up, when I installed and worked with package in one repo clone (that contain a `conanfile.txt`), then check out a new working copy of the same repo. `CONAN_USER_HOME` is machine-global on my system.

But I'll try to reproduce...
I finally got some time to review this effect. While I was not able to exactly reproduce the error I might have produced the situation that somehow will trigger it. For sure the reached state is highly inconsistent, as far as I'd say...

**First, my environment:**

* Windows 7 Enterprise, 64 Bit
* Python 3.6.3 64 Bit
* conan installed in virtual environment
* conan 0.30.1
* Packages (for debugging) served by built-in conan server on another machine

**What I am using conan for right now**

I am actually _not_ (yet) using it to manage library dependencies, but use it as a nice frontend to install development tools. I have packaged up a few tool dependencies, such that my conanfile looks like this:

```
[requires]
cmake/3.9.4@gen5/stable
ghs-arm/2015.1@gen5/stable
ghs-x86/2017.1@gen5/stable
ninja/1.8.2@gen5/stable
swegen/3.8.0@gen5/stable

[generators]
txt
virtualenv
```

So I am installing CMake, some compilers, ninja etc.

**Breaking consistency**

When I then do `conan install` the tool detects the dependencies and starts downloading packages. Since such tools are sometimes large (over 50 megs) users might interrupt with CTRL-C. The issue is then that (as far as I understand) that conan incorrectly assumes that package actually has been installed.

**Log**

Here is my activity log executed in a Windows console with the virtual environment active, where conan has been installed.

```
(venv) λ conan install                                                                                                                           
cmake/3.9.4@gen5/stable: Not found in local cache, looking in remotes...                                                                         
cmake/3.9.4@gen5/stable: Trying with 'mike'...                                                                                                   
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 58B/58B                                                                                     
Downloading conanfile.py                                                                                                                         
[==================================================] 618B/618B                                                                                   
ghs-arm/2015.1@gen5/stable: Not found in local cache, looking in remotes...                                                                      
ghs-arm/2015.1@gen5/stable: Trying with 'mike'...                                                                                                
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 58B/58B                                                                                     
Downloading conanfile.py                                                                                                                         
[==================================================] 499B/499B                                                                                   
ghs-x86/2017.1@gen5/stable: Not found in local cache, looking in remotes...                                                                      
ghs-x86/2017.1@gen5/stable: Trying with 'mike'...                                                                                                
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 58B/58B                                                                                     
Downloading conanfile.py                                                                                                                         
[==================================================] 507B/507B                                                                                   
ninja/1.8.2@gen5/stable: Not found in local cache, looking in remotes...                                                                         
ninja/1.8.2@gen5/stable: Trying with 'mike'...                                                                                                   
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 58B/58B                                                                                     
Downloading conanfile.py                                                                                                                         
[==================================================] 529B/529B                                                                                   
swegen/3.8.0@gen5/stable: Not found in local cache, looking in remotes...                                                                        
swegen/3.8.0@gen5/stable: Trying with 'mike'...                                                                                                  
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 58B/58B                                                                                     
Downloading conanfile.py                                                                                                                         
[==================================================] 644B/644B                                                                                   
PROJECT: Installing W:\Temp\conandbg                                                                                                             
Requirements                                                                                                                                     
    cmake/3.9.4@gen5/stable from mike                                                                                                            
    ghs-arm/2015.1@gen5/stable from mike                                                                                                         
    ghs-x86/2017.1@gen5/stable from mike                                                                                                         
    ninja/1.8.2@gen5/stable from mike                                                                                                            
    swegen/3.8.0@gen5/stable from mike                                                                                                           
Packages                                                                                                                                         
    cmake/3.9.4@gen5/stable:ca33edce272a279b24f87dc0d4cf5bbdcffbc187                                                                             
    ghs-arm/2015.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ghs-x86/2017.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ninja/1.8.2@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
    swegen/3.8.0@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                            
                                                                                                                                                 
PROJECT: Retrieving package ca33edce272a279b24f87dc0d4cf5bbdcffbc187                                                                             
cmake/3.9.4@gen5/stable: Looking for package ca33edce272a279b24f87dc0d4cf5bbdcffbc187 in remote 'mike'                                           
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 416.5KB/416.5KB                                                                             
Downloading conaninfo.txt                                                                                                                        
[==================================================] 219B/219B                                                                                   
Downloading conan_package.tgz                                                                                                                    
[==========================                        ] 10.8MB/20.8MB    You pressed Ctrl+C!                                                        
```

So here the installation started and I hit CTRL-C during the first package download (CMake). Then I just redo this. Note that the interrupted downloads are _not_ reattempted, but conan claims those packages are "Already installed":

```
(venv) λ conan install                                                                                                                           
PROJECT: Installing W:\Temp\conandbg                                                                                                             
Requirements                                                                                                                                     
    cmake/3.9.4@gen5/stable from mike                                                                                                            
    ghs-arm/2015.1@gen5/stable from mike                                                                                                         
    ghs-x86/2017.1@gen5/stable from mike                                                                                                         
    ninja/1.8.2@gen5/stable from mike                                                                                                            
    swegen/3.8.0@gen5/stable from mike                                                                                                           
Packages                                                                                                                                         
    cmake/3.9.4@gen5/stable:ca33edce272a279b24f87dc0d4cf5bbdcffbc187                                                                             
    ghs-arm/2015.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ghs-x86/2017.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ninja/1.8.2@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
    swegen/3.8.0@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                            
                                                                                                                                                 
cmake/3.9.4@gen5/stable: Already installed!                                                                                                      
PROJECT: Retrieving package 3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
ghs-arm/2015.1@gen5/stable: Looking for package 3475bd55b91ae904ac96fde0f106a136ab951a5e in remote 'mike'                                        
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 560.2KB/560.2KB                                                                             
Downloading conaninfo.txt                                                                                                                        
[==================================================] 187B/187B                                                                                   
Downloading conan_package.tgz                                                                                                                    
[=======                                           ] 104.4MB/745.5MB  You pressed Ctrl+C!                                                        
                                                                                                                                                 
W:\Temp\conandbg                                                                                                                                 
(venv) λ conan install                                                                                                                           
PROJECT: Installing W:\Temp\conandbg                                                                                                             
Requirements                                                                                                                                     
    cmake/3.9.4@gen5/stable from mike                                                                                                            
    ghs-arm/2015.1@gen5/stable from mike                                                                                                         
    ghs-x86/2017.1@gen5/stable from mike                                                                                                         
    ninja/1.8.2@gen5/stable from mike                                                                                                            
    swegen/3.8.0@gen5/stable from mike                                                                                                           
Packages                                                                                                                                         
    cmake/3.9.4@gen5/stable:ca33edce272a279b24f87dc0d4cf5bbdcffbc187                                                                             
    ghs-arm/2015.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ghs-x86/2017.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ninja/1.8.2@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
    swegen/3.8.0@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                            
                                                                                                                                                 
cmake/3.9.4@gen5/stable: Already installed!                                                                                                      
ghs-arm/2015.1@gen5/stable: Already installed!                                                                                                   
PROJECT: Retrieving package 3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
ghs-x86/2017.1@gen5/stable: Looking for package 3475bd55b91ae904ac96fde0f106a136ab951a5e in remote 'mike'                                        
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 90.7KB/90.7KB                                                                               
Downloading conaninfo.txt                                                                                                                        
[==================================================] 187B/187B                                                                                   
Downloading conan_package.tgz                                                                                                                    
[==                                                ] 4.4MB/109.9MB    You pressed Ctrl+C!                                                        
                                                                                                                                                 
W:\Temp\conandbg                                                                                                                                 
(venv) λ conan install                                                                                                                           
PROJECT: Installing W:\Temp\conandbg                                                                                                             
Requirements                                                                                                                                     
    cmake/3.9.4@gen5/stable from mike                                                                                                            
    ghs-arm/2015.1@gen5/stable from mike                                                                                                         
    ghs-x86/2017.1@gen5/stable from mike                                                                                                         
    ninja/1.8.2@gen5/stable from mike                                                                                                            
    swegen/3.8.0@gen5/stable from mike                                                                                                           
Packages                                                                                                                                         
    cmake/3.9.4@gen5/stable:ca33edce272a279b24f87dc0d4cf5bbdcffbc187                                                                             
    ghs-arm/2015.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ghs-x86/2017.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ninja/1.8.2@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
    swegen/3.8.0@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                            
                                                                                                                                                 
cmake/3.9.4@gen5/stable: Already installed!                                                                                                      
ghs-arm/2015.1@gen5/stable: Already installed!                                                                                                   
ghs-x86/2017.1@gen5/stable: Already installed!                                                                                                   
PROJECT: Retrieving package 3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
ninja/1.8.2@gen5/stable: Looking for package 3475bd55b91ae904ac96fde0f106a136ab951a5e in remote 'mike'                                           
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 103B/103B                                                                                   
Downloading conaninfo.txt                                                                                                                        
[==================================================] 187B/187B                                                                                   
Downloading conan_package.tgz                                                                                                                    
[==================================================] 241.3KB/241.3KB                                                                             
ninja/1.8.2@gen5/stable: Package installed 3475bd55b91ae904ac96fde0f106a136ab951a5e                                                              
PROJECT: Retrieving package 3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
swegen/3.8.0@gen5/stable: Looking for package 3475bd55b91ae904ac96fde0f106a136ab951a5e in remote 'mike'                                          
Downloading conanmanifest.txt                                                                                                                    
[==================================================] 2.9KB/2.9KB                                                                                 
Downloading conaninfo.txt                                                                                                                        
[==================================================] 187B/187B                                                                                   
Downloading conan_package.tgz                                                                                                                    
[=================                                 ] 5.0MB/14.4MB     You pressed Ctrl+C!                                                        
                                                                                                                                                 
W:\Temp\conandbg                                                                                                                                 
(venv) λ conan install                                                                                                                           
PROJECT: Installing W:\Temp\conandbg                                                                                                             
Requirements                                                                                                                                     
    cmake/3.9.4@gen5/stable from mike                                                                                                            
    ghs-arm/2015.1@gen5/stable from mike                                                                                                         
    ghs-x86/2017.1@gen5/stable from mike                                                                                                         
    ninja/1.8.2@gen5/stable from mike                                                                                                            
    swegen/3.8.0@gen5/stable from mike                                                                                                           
Packages                                                                                                                                         
    cmake/3.9.4@gen5/stable:ca33edce272a279b24f87dc0d4cf5bbdcffbc187                                                                             
    ghs-arm/2015.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ghs-x86/2017.1@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                          
    ninja/1.8.2@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                             
    swegen/3.8.0@gen5/stable:3475bd55b91ae904ac96fde0f106a136ab951a5e                                                                            
                                                                                                                                                 
cmake/3.9.4@gen5/stable: Already installed!                                                                                                      
ghs-arm/2015.1@gen5/stable: Already installed!                                                                                                   
ghs-x86/2017.1@gen5/stable: Already installed!                                                                                                   
ninja/1.8.2@gen5/stable: Already installed!                                                                                                      
swegen/3.8.0@gen5/stable: Already installed!                                                                                                     
PROJECT: Generator txt created conanbuildinfo.txt                                                                                                
PROJECT: Generator virtualenv created activate.bat                                                                                               
PROJECT: Generator virtualenv created deactivate.bat                                                                                             
PROJECT: Generator virtualenv created activate.ps1                                                                                               
PROJECT: Generator virtualenv created deactivate.ps1                                                                                             
PROJECT: Generated conaninfo.txt                                                                                                                 
```

Eventually I end up with all dependencies apparently installed, but in fact, they are not. This for instance is the state of one of the package directories:

```
(venv) λ tree /F ghs-arm
W:\TEMP\CONANDBG\.CONAN\DATA\GHS-ARM
└───2015.1
    └───gen5
        └───stable
            │   rw.count
            │   rw.count.lock
            │
            ├───export
            │       conanfile.py
            │       conanmanifest.txt
            │
            ├───locks
            │       3475bd55b91ae904ac96fde0f106a136ab951a5e
            │
            └───package
                └───3475bd55b91ae904ac96fde0f106a136ab951a5e
                        conaninfo.txt
                        conanmanifest.txt
                        conan_package.tgz
```

Some part of the package archive is there but that's it. It has not been completed, nor unpacked. Are those lock files and folders expected?

Please let me know if you need more details, I hope this helps analyzing the issue. Leaving users in this state is really problematic in my use-case I developers _will_ interrupt the download if it takes too long from what they initially expect.

Once in this situation you have to `conan remove -f *` to start over.

Thanks!
Thanks very much for trying this, and all this detailed feedaback, it is really useful!

The first thing is that the good news are that the locks are not causing too much trouble here. This is one thing, if they were, you will see locks, a warning message and a request to remove things from the cache.

What you are experiencing is a known (different) issue (check https://github.com/conan-io/conan/issues/1412, some previous work here: https://github.com/conan-io/conan/pull/1752), right now there are no checks for package completion. 

So if things are interrupted, it is true that the cache can be inconsistent, and the current approach is to clean the cache. I think we want 2 things:
- Add this to the FAQ
- Check this as possible enhancement to the build and package folders. 

Thanks for raising this issue and your feedback
The other issue I was looking for: https://github.com/conan-io/conan/issues/1003
Hello!

I just stumbled upon a locking issue in our CI which was caused by a `conan install` being interrupted by a SIGTERM. After some investigation I figured out the scenario which can lead to such a deadlock.

The problem arise when conan gets a SIGTERM during a package build as it hold the `conanfile_read_lock`. In such a case it will leave behind a reader count > 0. On further invocation, it will try to get a `conanfile_write_lock` to prepare the build and fail because the reader count is not 0. All this happens in `conans/client/installer.py` in `ConanInstaller._build_package()`.

The current implementation of `conans.util.locks.ReadLock` does not guarantee that the reader count will be decreased when a SIGTERM is received as python will not catch it and of course it won't dispatch an exception so the `__exit__` of the lock is not called. I'm attaching a patch to fix that. It works fine on Linux but I haven't tested it on any other platform. The present implementation will restore the previous handler and resend the signal so that the process really gets terminated.

Note that the `WriteLock` suffer from the same problem but if a write lock is held while SIGTERMed, it's not obvious what needs to be done to reach a consistent state before releasing the lock. Raising an exception to properly cleanup and terminate would probably be the best solution (also for the `ReadLock`). However it requires more knowledge of the inner working of conan for me to propose a patch.

[patch_readlock_handle_sigterm.txt](https://github.com/conan-io/conan/files/2072435/patch_readlock_handle_sigterm.txt)


Hi @odormond !

Thanks for the report.

The signals are currently registered in command.py:

```python
       def ctrl_c_handler(_, __):
            sys.exit(3)

        def ctrl_break_handler(_, __):
            sys.exit(4)

        signal.signal(signal.SIGINT, ctrl_c_handler)
        ...
```
They translate to a sys.exit, which in turn will raising SystemExit, so it will give time for locks to close properly. I think it could be enough to just set the SIGTERM handler too here. Wdyt @lasote?

Also, just in case, you have the ``conan remove --locks`` command to recover easily from this condition.


Resolving patch:
diff --git a/conans/client/command.py b/conans/client/command.py
--- a/conans/client/command.py
+++ b/conans/client/command.py
@@ -1305,11 +1305,17 @@ def ctrl_c_handler(_, __):
             print('You pressed Ctrl+C!')
             sys.exit(3)
 
+        def sigterm_handler(_, __):
+            print('Received SIGTERM!')
+            sys.exit(5)
+
         def ctrl_break_handler(_, __):
             print('You pressed Ctrl+Break!')
             sys.exit(4)
 
         signal.signal(signal.SIGINT, ctrl_c_handler)
+        signal.signal(signal.SIGTERM, sigterm_handler)
+
         if sys.platform == 'win32':
             signal.signal(signal.SIGBREAK, ctrl_break_handler)
         error = command.run(args)


Test patch: