Basic operations do not work in 1.1.0 with uWSGI+Flask

lebionick: <p>Came across the problem, that in pytorch <strong>1.1.0</strong> with uwsgi+flask on cpu even <code>torch.cat</code> does not work (everything just <em>freezes without errors</em>). I determined that problem is in <strong>uwsgi/flask</strong>, because in the same environment I was able to do the same operations without any issues. There was <strong>no</strong> such problem in previous versions of pytorch. As far as I understand, I am dealing with <em>multiprocessing</em> artifacts…<br>
Nevertheless I <strong>found</strong> the solution and it was simple:</p>
<pre><code class="lang-python">app = flask.Flask(__name__)
segmentator = None

@app.before_first_request
def load_segmentator():
    global segmentator
    segmentator = Segmentator()
</code></pre>
<p>where <code>Segmentator</code> is a class with pytorch’s <code>nn.Module</code>, which loads weights in <code>__init__</code><br>
Hope, it will help somebody.</p>
<p>P.S. If someone explains what is going on here, will be grateful.</p>

ijnf: <p>I’m facing the same issue. i tried a similar solution but still doesn’t seem to work. Any more information about this would be appreciated</p>

lebionick: <p>To be honest I’m not really understand why it worked and cannot say much more in terms of code.<br>
Here is my “code” inside <code>Segmentator</code>:</p>
<pre><code class="lang-auto">class Segmentator:
    def __init__(self, path):
        print('loading model')
        loaded_model = UNetWithResnet50Encoder(9)
        print('loading weights')
        checkpoint = torch.load(path, map_location='cpu')
        print('loading checkpoint')
        loaded_model.load_state_dict(checkpoint['model'])
        loaded_model.eval()
        print('loaded!')
        self.model = loaded_model

    def process(img_url):
        # loads image into RAM, preprocesses it,
        # passes trough self.model, and postprocesses output
        pass
</code></pre>

ijnf: <p><a class="mention" href="/u/lebionick">@lebionick</a> thanks for the info. I actually have the exact same project structure as your code. <code>Segmentor</code> just named differently <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I had another solution for another app but I changed the solution to be exactly like yours with the <code>before_first_request</code> still to no success. I will report back if I find out more about this.</p>

ijnf: <p>ok it seems <code>load_state_dict</code> and other operations in <code>__init__</code> when run after <code>flask.Flask(__name__)</code> cause pytorch operations done in requests to hang forever. Not sure about the cause but maybe this info could help someone</p>

Khan1: <p>Try  <a href="https://github.com/craigsidcarlson/PytorchFlaskApp/blob/master/app/application.py" rel="nofollow noopener">this</a></p>

keng000: <p>Don’t you running the flask app with uWSGI’s “preforking worker mode” that is a basic config?<br>
Try <code>lazy-apps</code> mode that each worker will load the trained model for them self and not share with others.<br>
It works in my environment even I load model in the global.</p>
<p>command<br>
<code>uwsgi --lazy-app </code></p>
<p>or ini file</p>
<pre><code class="lang-auto">[uwsgi]
lazy-apps = true
</code></pre>
<p>minimum reproduction<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/keng000/uwsgi-flask-ml" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21191288?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/keng000/uwsgi-flask-ml" target="_blank" rel="nofollow noopener">keng000/uwsgi-flask-ml</a></h3>

<p>Contribute to keng000/uwsgi-flask-ml development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

randomwalk10: <p>By setting lazy-apps to be true, it solves the problem I had with flask+uwsgi+pytorch deployment. Thx.</p>

af_luther: <p>I found this thread really interesting because I am working on scaling my inference server (which uses uWSGI + flask + PyTorch on AWS with Elastic Inference), and when I increased the # of processes recently I came across some intermittent issues:</p>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  [enforce fail at inline_container.cc:316] . PytorchStreamWriter failed writing central directory: file write failed</p>
<p>I couldn’t find any other references to this sort of error, but I am guessing it could be some kind of concurrency issue as I increased the number of processes.</p>

alexandru-dinu: <p>I found a different solution that works in my case and still maintains the default fork model of uWSGI (i.e. without <code>lazy-apps</code>). Enabling <code>lazy-apps</code> may not desirable because worker processes will not share resources with their parent (e.g. read-only resources such as models used for inference).</p>
<p>The idea is to share <strong>only</strong> the <code>state_dict</code> between processes, and each worker process will do <code>model.load_state_dict(global_state)</code> independently, but only once, during the first request. This is opposed to doing <code>load_state_dict</code> in the parent process and sharing this with the worker processes.</p>
<p>I managed to replicate this behavior in this <a href="https://stackoverflow.com/questions/67836858/child-process-hangs-when-performing-inference-with-pytorch-model" rel="noopener nofollow ugc">StackOverflow post</a>.</p>
<p>I hope this helps others as well!</p>

mneilly: <p>I just ran into a similar issue running with pytorch and uwsgi in a container. This issue (<a href="https://github.com/pytorch/pytorch/issues/50669" class="inline-onebox" rel="noopener nofollow ugc">Launching two processes causes hanging · Issue #50669 · pytorch/pytorch · GitHub</a>) indicates that using LD_PRELOAD to use Intel’s OMP instead of libgomp could avoid the issue which is in libgomp. On Debian Bullseye I was able to install libomp-dev-11 and <code>LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libiomp5.so /usr/bin/uwsgi --ini /uwsgi.ini --uid www-data --enable-threads</code> to workaround the issue.</p>
<p>FWIW - I created a github repo to demonstrate the issue and workaround here: <a href="https://github.com/mneilly/pytorch-libgomp-hang-and-workaround-example" class="inline-onebox" rel="noopener nofollow ugc">GitHub - mneilly/pytorch-libgomp-hang-and-workaround-example: This is an example of the libgomp hang issue encountered in pytorch</a></p>