Repository: zulip/zulip

Issue:
Support restarting the server without rejecting any requests
In theory, it should be possible with uwsgi and its `master=true` setting to restart the server with 0 requests being rejected due to the service being down (the approach seems to be the obvious thing of queuing requests in the socket until the new processes are up).  I tried this briefly with our supervisord and ran into problems where it would just fail to restart, so some investigation is required into how to do this properly.




Discussion:
@zulipbot claim
@timabbott,
Supervisord doesn't allow to reload applications. It just contains the interface for restarting programs through stop-start command. It means, that when we send HUP signal to uWSGI through supervisor it start reloading child processes, but the master process still alive. Supervisord is waiting for application stopping at this moment and kills the master process after `stopwaitsec` period. This behavior is the cause of errors "502 bad gateway" on restarting `zulip-django` application with supervisord. 
For graceful reloading, we can enable The master FIFO mode (http://uwsgi-docs.readthedocs.io/en/latest/MasterFIFO.html) and write `r` to master fifo file. It will allow us to reload uWSGI workers without requests rejecting. 
There are several additional methods to reload uWSGI workers through fifo file. 

One of them it is using `lazy-app` mode. In this mode, each worker uses the own copy of the application(don't share a memory from master).  It allows wait for running workers and then restarts each of them. But also it will use more memory than in simple mode. 

Another way is chain reloading. It also uses `lasy-app` mode with writing `c` to fifo file. In this mode, uWSGI restarts one worker at the time and do not reload another while previous will be not ready to receive requests.
Hello @kkanahin, you claimed this issue to work on it, but this issue and any referenced pull requests haven't been updated for a week. Are you still working on this issue?

If so, please update this issue by leaving a comment on this issue to let me know that you're still working on it. Otherwise, I'll automatically remove you from this issue in 3 days.

If you've decided to work on something else, simply comment `@zulipbot abandon` so that someone else can claim it and continue from where you left off.

Thank you for your valuable contributions to Zulip!

@timabbott Could you please review my investigations.
Col, I think my conclusion from that is that this is somewhat tricky to do with supervisord and potentially something we should defer until later.
I think we need separate tools for reloading services. And allow supervisor just controls stop start commands.
Hello @kkanahin, you have been unassigned from this issue because you have not updated this issue or any referenced pull requests for over ten days.

You can reclaim this issue or claim any other issue by commenting `@zulipbot claim` on that issue.

Thanks for your contributions, and hope to see you again soon!


Resolving patch:
diff --git a/scripts/lib/zulip_tools.py b/scripts/lib/zulip_tools.py
--- a/scripts/lib/zulip_tools.py
+++ b/scripts/lib/zulip_tools.py
@@ -623,7 +623,7 @@ def list_supervisor_processes(*args: str) -> List[str]:
         universal_newlines=True,
         stdout=subprocess.PIPE,
     )
-    # `supercisorctl status` returns 3 if any are stopped, which is
+    # `supervisorctl status` returns 3 if any are stopped, which is
     # fine here; and exit code 4 is for no such process, which is
     # handled below.
     if worker_status.returncode not in (0, 3, 4):


Test patch: