Repository: evennia/evennia

Issue:
Very rarely server hangs indefinitely on 'evennia reload' from console
#### Brief summary of issue / Description of requested feature:
This is a very rare bug that I have no idea how to reproduce. It's only happened 3 times now out of reloading hundreds of times. What happens is when starting a reload, the only output you'll receive is 'received SIGINT, shutting down'. And then absolutely nothing else will happen - you won't be told ports are closing, no attempt to restart, etc. Attempting to do evennia start will then tell you that the server/portal process is already running, while stopping will tell you that both are stopped, but you'll be informed the server is still running when you try to start. The server will continue until it's wiped with kill -9, and the pid file is removed. It occurs to me that I've never seen this happen with reloading from inside the game, only with attempting to reload with 'evennia reload' from the console.


#### Steps to reproduce the issue / Reasons for adding feature:

1. 'evennia reload' and then maybe, just maybe, you might get very unlucky. That's the best I have to go on at the moment.

#### Error output / Expected result of feature
'Server received SIGINT, shutting down', and then nothing.

#### Extra information, such as Evennia revision/repo/branch, operating system and ideas for how to solve / implement:

The only reason I was thinking of this was all the craziness with the Windows reload process, and wondering if there was any strangeness happening with specifically the console version of reload. Since I have no idea how to reproduce it and it seems very rare, I just wanted to mention it here in case anyone else runs into it and might be able to shed more light on what might cause it to happen.

Very rarely server hangs indefinitely on 'evennia reload' from console
#### Brief summary of issue / Description of requested feature:
This is a very rare bug that I have no idea how to reproduce. It's only happened 3 times now out of reloading hundreds of times. What happens is when starting a reload, the only output you'll receive is 'received SIGINT, shutting down'. And then absolutely nothing else will happen - you won't be told ports are closing, no attempt to restart, etc. Attempting to do evennia start will then tell you that the server/portal process is already running, while stopping will tell you that both are stopped, but you'll be informed the server is still running when you try to start. The server will continue until it's wiped with kill -9, and the pid file is removed. It occurs to me that I've never seen this happen with reloading from inside the game, only with attempting to reload with 'evennia reload' from the console.


#### Steps to reproduce the issue / Reasons for adding feature:

1. 'evennia reload' and then maybe, just maybe, you might get very unlucky. That's the best I have to go on at the moment.

#### Error output / Expected result of feature
'Server received SIGINT, shutting down', and then nothing.

#### Extra information, such as Evennia revision/repo/branch, operating system and ideas for how to solve / implement:

The only reason I was thinking of this was all the craziness with the Windows reload process, and wondering if there was any strangeness happening with specifically the console version of reload. Since I have no idea how to reproduce it and it seems very rare, I just wanted to mention it here in case anyone else runs into it and might be able to shed more light on what might cause it to happen.



Discussion:
I don't know what could cause this on Linux. Having reloaded Evennia *thousands* of times, I never encounteted a case when the reloader fails -because the reload mechanism I (as far as I know) very robust under an OS with proper signal handling. One thing I do know is that the reload may take longer or shorter at times, depending on how much cleanup and objets with pre-reload hooks are cached. Only thing I can think of is the reload somehow clashing with the cache flush; might be worth checking if there is a correlation between the flush and the times the reload fails.
@TehomCD Any more data on this with the logging statements we discussed in IRC?
Had it finally happen again after putting in the logging, just now.
```(arx) tehom@arxhost:~/arx/arx$ 2016-12-14T07:07:02-0500 [stdout#info] Beginning shutdown.
2016-12-14T07:07:02-0500 [stdout#info] Setting restart mode.
2016-12-14T07:07:02-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2016-12-14T07:07:02-0500 [stdout#info] Handling ObjectDB cached instances.
2016-12-14T07:07:02-0500 [stdout#info] Handling PlayerDB cached instances.
2016-12-14T07:07:02-0500 [stdout#info] Handling ScriptDB cached instances.
2016-12-14T07:07:03-0500 [stdout#info] Syncing sessions.
2016-12-14T07:07:03-0500 [stdout#info] Calling server reload hook.
2016-12-14T07:07:03-0500 [stdout#info] Saving MONITOR_HANDLER.
2016-12-14T07:07:03-0500 [stdout#info] Beginning shutdown.
```

And that's the end of my print statements, though it looks like I should have added more to be thorough. The last thing seen is before returning the None value in the deferred from the callLater statement. 

After removing the server.pid and trying to start it up again, which got an error about the port already being used, since the twisted process hadn't been killed yet, I got the notification of evennia runner being killed, so it seems to have thought that the evennia_runner was still alive. I then just killed the stuck server process and started it up, and players just saw it as a frighteningly long reload - the portal still always stays alive throughout the process.
@TehomCD So what is missing from that output is just where those strings are printed. So exactly where is "Beginning shutdown" printed?
```
 @defer.inlineCallbacks
    def shutdown(self, mode=None, _reactor_stopping=False):
        """
        Shuts down the server from inside it.

        mode - sets the server restart mode.
               'reload' - server restarts, no "persistent" scripts
                          are stopped, at_reload hooks called.
               'reset' - server restarts, non-persistent scripts stopped,
                         at_shutdown hooks called but sessions will not
                         be disconnected.
               'shutdown' - like reset, but server will not auto-restart.
               None - keep currently set flag from flag file.
        _reactor_stopping - this is set if server is stopped by a kill
                            command OR this method was already called
                             once - in both cases the reactor is
                             dead/stopping already.
        """
        print("Beginning shutdown.")
        if _reactor_stopping and hasattr(self, "shutdown_complete"):
            # this means we have already passed through this method
            # once; we don't need to run the shutdown procedure again.
            defer.returnValue(None)
        print("Setting restart mode.")
        mode = self.set_restart_mode(mode)

        from evennia.objects.models import ObjectDB
        #from evennia.players.models import PlayerDB
        from evennia.server.models import ServerConfig

        if mode == 'reload':
            # call restart hooks
            print("Mode is reload. Setting ServerConfig.objects.conf")
            ServerConfig.objects.conf("server_restart_mode", "reload")
            print("Handling ObjectDB cached instances.")
            yield [o.at_server_reload() for o in ObjectDB.get_all_cached_instances()]
            print("Handling PlayerDB cached instances.")
            yield [p.at_server_reload() for p in PlayerDB.get_all_cached_instances()]
            print("Handling ScriptDB cached instances.")
            yield [(s.pause(manual_pause=False), s.at_server_reload()) for s in ScriptDB.get_all_cached_instances() if s.is_active]
            print("Syncing sessions.")
            yield self.sessions.all_sessions_portal_sync()
            print("Calling server reload hook.")
            self.at_server_reload_stop()
            # only save monitor state on reload, not on shutdown/reset
            from evennia.scripts.monitorhandler import MONITOR_HANDLER
            print("Saving MONITOR_HANDLER.")
            MONITOR_HANDLER.save()
```

It looks like I omitted the first line of the log, 'Beginning shutdown' was also printed at the start. It appears at the start of the method, so the second one, which made the last print statement, seems to be from the callback.
Hm, so we are missing output tests after the monitorhandler save then, from https://github.com/evennia/evennia/blob/master/evennia/server/server.py#L380 and onward. The second call of "Beginning shutdown" is also OK, this is a catch-all because the reactor will react to the shut-down signal on its own). 
But this at least tells us that the calling of all on-object hooks seem to have been finished correctly. What remains are only the tickerhandler resync and the actual stopping of the reactor which is not supposed to be any problem... 

From what you write the evennia runner is not dead and in fact the Server's ports are not even closeing which rules out some sort of problem with the runner going out of scope or signals not passing correctly. This seems to be due to the Twisted reactor not correctly shutting down when sent an external signal but locking up and staying in some sort of limbo. 

It would be good if you could put prints in the small remainder of the `shutdown` method just on the off-chance we can spot where the lock-up is. Especially at the tickerhandler and before/after the actual dcall to stop the reactor would be useful. 
Actually, I would suspect the reactor.stop() actually works too, since this is what causes Twisted to send the responds to and is causing that second call of `shutdown` (the one causing the second "Beginning shutdown"). That signal is setup [here](https://github.com/evennia/evennia/blob/master/evennia/server/server.py#L171).  Since there is no more output after that, I suspect the `shutdown` method goes into the if statement. In which case we won't be able to pin-point it since it then goes into `defer.returnValue(None)`.
I added the print statements to the rest of the shutdown method, so hopefully next time we'll be able to ensure the rest of that finishes correctly. If there's anything else you think that might help to have prints at, let me know and I'll add 'em.
I wonder if `defer.returnValue` would lock up if the reactor has already managed to shut down before it is called. If so it might explain why it happens with irregularity - most often it gets called while the reactor is still in the process of shutting down (just how fast this happens depends on the speed of the signal processing) but some times the reactor is already gone. Not sure.  
Happened again, and the last statement to execute was my print statement before returning the deferred, so it's possible that may be the culprit for some reason.
```
2016-12-18T19:55:37-0500 [stdout#info] Beginning shutdown.
2016-12-18T19:55:37-0500 [stdout#info] Setting restart mode.
2016-12-18T19:55:37-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2016-12-18T19:55:37-0500 [stdout#info] Handling ObjectDB cached instances.
2016-12-18T19:55:37-0500 [stdout#info] Handling PlayerDB cached instances.
2016-12-18T19:55:37-0500 [stdout#info] Handling ScriptDB cached instances.
2016-12-18T19:55:37-0500 [stdout#info] Syncing sessions.
2016-12-18T19:55:37-0500 [stdout#info] Calling server reload hook.
2016-12-18T19:55:37-0500 [stdout#info] Saving MONITOR_HANDLER.
2016-12-18T19:55:37-0500 [stdout#info] Saving TICKER_HANDLER.
2016-12-18T19:55:37-0500 [stdout#info] Calling server_stop hook.
2016-12-18T19:55:37-0500 [stdout#info] Before reactor.callLater
2016-12-18T19:55:37-0500 [stdout#info] End of shutdown method.
2016-12-18T19:55:37-0500 [stdout#info] Beginning shutdown.
2016-12-18T19:55:37-0500 [stdout#info] Inside deferred, returnValue(None)
```

One question - what makes the portal print out '...' while waiting for the server to restart? I noticed it doesn't do that until I kill the hung server process, which then prints out my notification that the evennia runner has been killed. So wherever in the shutdown/restart process it hangs, it stops before the point where it starts printing out ellipses.
The problem seems to have become more frequent with time or game activity, so I'm trying to think of anything related to that - had it hang two times in quick succession recently. I wonder if logging could somehow be to blame? My portal and http logs tend to rotate a few times a day, and when I forget to clean them out for a few days I start having things like `portal.log.27` or whatever. I'll try to diligently remove them on a daily basis and see if the problem goes back to being much more rare.
As a contrast for clarity, here's what a normal reload has that it's missing from the point it hangs:
```
2017-01-07T00:49:12-0500 [-] (TCP Port 5001 Closed)
2017-01-07T00:49:12-0500 [-] (TCP Port 5000 Closed)
2017-01-07T00:49:12-0500 [-] Main loop terminated.
2017-01-07T00:49:12-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.
Server restarting ...
--------------------------------------------------
 Arx Server (0.6.0 (rev 5b06107)) started.
  amp (to Portal): 5000
  webserver: 5001
--------------------------------------------------
2017-01-07T00:49:13-0500 [-] Loading /home/tehom/arx/evennia/evennia/server/server.py...
2017-01-07T00:49:14-0500 [-] Loaded.
2017-01-07T00:49:14-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 16.4.0 (/home/tehom/arx/bin/python2 2.7.12) starting up.
2017-01-07T00:49:14-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2017-01-07T00:49:14-0500 [-] AmpServerFactory starting on 5000
2017-01-07T00:49:14-0500 [-] Website starting on 5001
2017-01-07T00:49:15-0500 [evennia.server.amp.AmpServerFactory] AMPProtocol connection established (HOST:IPv4Address(TCP, '127.0.0.1', 5000) PEER:IPv4Address(TCP, '127.0.0.1', 45424))
```
The ports remain in use until the server process is killed, at which point the evennia runner is also killed as well. Until that point, the Evennia Runner is still running, but doesn't break out of its reload loop: it hasn't yet been sent the signal to restart the server.

I think I'll try removing the `defer.returnValue(None)` statement entirely, just having it do an if/else check instead, and see if that changes anything.
No luck - still hung without with the `defer.returnValue(None)` statement: that block now only prints out the statement about it being inside the deferred block, and does nothing else. Here's the most recent log (snipped messages about staticfiles for brevity), including what happens after I kill the process and start it back up again.
```
2017-01-08T19:22:53-0500 [stdout#info] Beginning shutdown.
2017-01-08T19:22:53-0500 [stdout#info] Setting restart mode.
2017-01-08T19:22:53-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2017-01-08T19:22:53-0500 [stdout#info] Handling ObjectDB cached instances.
2017-01-08T19:22:53-0500 [stdout#info] Handling PlayerDB cached instances.
2017-01-08T19:22:53-0500 [stdout#info] Handling ScriptDB cached instances.
2017-01-08T19:22:54-0500 [stdout#info] Syncing sessions.
2017-01-08T19:22:54-0500 [stdout#info] Calling server reload hook.
2017-01-08T19:22:54-0500 [stdout#info] Saving MONITOR_HANDLER.
2017-01-08T19:22:54-0500 [stdout#info] Saving TICKER_HANDLER.
2017-01-08T19:22:54-0500 [stdout#info] Calling server_stop hook.
2017-01-08T19:22:54-0500 [stdout#info] Before reactor.callLater
2017-01-08T19:22:54-0500 [stdout#info] End of shutdown method.
2017-01-08T19:22:54-0500 [stdout#info] Beginning shutdown.
2017-01-08T19:22:54-0500 [stdout#info] Inside deferred, returnValue(None)

(arx) tehom@arxhost:~/arx/arx$ evennia start

Evennia Server is already running as process 19023. Not restarted.

Evennia Portal is already running as process 7674. Not restarted.
(arx) tehom@arxhost:~/arx/arx$ kill -9 19023
(arx) tehom@arxhost:~/arx/arx$ message is server_stopped, int(rc) is -9, get_restart_mode is reload: 
Evennia runner has been killed!

(arx) tehom@arxhost:~/arx/arx$ cd server
(arx) tehom@arxhost:~/arx/arx/server$ rm server.pid
(arx) tehom@arxhost:~/arx/arx/server$ cd ..
(arx) tehom@arxhost:~/arx/arx$ evennia start

Starting Evennia Server (output to stdout).

Evennia Portal is already running as process 7674. Not restarted.
(arx) tehom@arxhost:~/arx/arx$ --------------------------------------------------
 Arx Server (0.6.0 (rev 6de75ea)) started.
  amp (to Portal): 5000
  webserver: 5001
--------------------------------------------------
2017-01-08T19:23:26-0500 [-] Loading /home/tehom/arx/evennia/evennia/server/server.py...
2017-01-08T19:23:27-0500 [-] Loaded.
2017-01-08T19:23:27-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 16.4.0 (/home/tehom/arx/bin/python2 2.7.12) starting up.
2017-01-08T19:23:27-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2017-01-08T19:23:27-0500 [-] AmpServerFactory starting on 5000
2017-01-08T19:23:27-0500 [-] Website starting on 5001
2017-01-08T19:23:28-0500 [evennia.server.amp.AmpServerFactory] AMPProtocol connection established (HOST:IPv4Address(TCP, '127.0.0.1', 5000) PEER:IPv4Address(TCP, '127.0.0.1', 32802))
```
Going to try changing the callLater for reactor.stop() to be a 1 second delay rather than 0, in case there might be any weird timing issues at play.
@TehomCD It's very strange that this keeps reappearing. If you monitor the processes in a separate window (say with `htop`, filtering on the twistd processes to only show them), could you maybe see if you see any differences in their behavior between a normal reload and one where it fails?
Good idea, I'll start taking a look at that. I haven't had a reload fail in the last few days since setting `callLater` to 1 rather than 0, but who knows if that's just a coincidence: I've gone weeks between a hang before.
Last night, I had what appeared to be a server hang during normal operation, but I have no idea what caused it - it wasn't prompted by an attempt to reload the server or anything else initiated on my part. After a few minutes, I tried to do `evennia reload` from the command line to see if it'd resolve it, and it froze during the attempt to reload. I have no idea if this is related to the hangs during `@reload` that I've seen previously, but I figured I might as well post where it stopped at:

```
Evennia: Server reload.
2017-01-18T11:23:48-0500 [AMPProtocol,0,127.0.0.1] Received SIGINT, shutting down.
(arx) tehom@arxhost:~/arx/arx$ 2017-01-18T11:23:53-0500 [stdout#info] Beginning shutdown.
2017-01-18T11:23:53-0500 [stdout#info] Setting restart mode.
2017-01-18T11:23:53-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2017-01-18T11:23:53-0500 [stdout#info] Handling ObjectDB cached instances.
2017-01-18T11:23:53-0500 [stdout#info] Handling PlayerDB cached instances.
2017-01-18T11:23:53-0500 [stdout#info] Handling ScriptDB cached instances.
2017-01-18T11:23:54-0500 [stdout#info] Syncing sessions.
```
So during this particular freeze, it happened at syncing sessions/before calling the server reload hook, in `yield self.sessions.all_sessions_portal_sync()`, calling either the AMP bridge or the portal process. I had to kill the portal process this time to get it unstuck, while the normal hangs I saw during a `@reload` required killing the server process. So probably something entirely different, but possibly related? Not sure.
Had another of the previous hangs with `callLater(1, reactor.stop)`. It did seem to become more infrequent with 1, but that might just be confirmation bias. I'm going to change it to `callLater(3, reactor.stop)` and see if there's any difference.
If you didn't see it, I should point out that the reload/startup hook's have changed last week, notably the calls of saving the monitor/tickerhandler in evennia/server/server.py. Just in case you need to update your printouts.
As for the problem itself; I wonder if this could be some sort of AMP issue at its core; if you say it freezes without a reload (assuming the problems are related) but the server is still running, there are only a few possible causes:

- AMP freezes or gets clogged up for some reason. This should be noticeable in the portal log. How long ago did you restart the Portal? Many upstream changes require Portal restart (just to rule that out).
- A network or database lockup. If the database slows down dramatically for some reason (e.g. due to filesystem errors) this *will* freeze Evennia, especially during reload. You should catch this with your server printouts though.
- An actual freeze or bug in the server that is swallowed by try...catch events and thus never reach the log. There are no known such catches in the core but it's not impossible. There could also be such in Arx' custom code I guess.

Yeah, I did merge those changes in and adjust my version of `server.py` with the name of the hook changing and so on. Last time I restarted the portal was when I had it freeze on me about two days ago. It's the first time I've seen the portal itself freeze that I can recall, and the behavior was markedly different than the crashes on a `reload` - just several minutes of the game failing to respond to commands, though I saw someone still able to log in, so the portal may have possibly still been working, then the portal hanging upon trying to `evennia reload` from the command line, ending in me having to kill the process.  That particular crash did seem to be AMP or portal related, since it froze right before `yield self.sessions.all_sessions_portal_sync()`, https://github.com/evennia/evennia/blob/master/evennia/server/server.py#L355 . This could have been an example of a database slowdown or network lockup crashing evennia during a reload, maybe?

Other than that, I haven't restarted the portal very regularly - I think it may have been a week or two before that since I last restarted it. Looking at the log, there's the occasional mysterious AMP errors that don't give any information, just an integer, but I kind of doubt they're related - I see those on my test machine too, and I've never had a freeze there.
While trying to investigate the lag issue I have with the latest commit, a player disconnected during a reload process and a server hang immediately followed. Leaving the log here as a possible cause:

```
2017-01-28T07:05:36-0500 [stdout#info] Beginning shutdown.
2017-01-28T07:05:36-0500 [stdout#info] Setting restart mode.
2017-01-28T07:05:36-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2017-01-28T07:05:36-0500 [stdout#info] Handling ObjectDB cached instances.
2017-01-28T07:05:36-0500 [stdout#info] Handling PlayerDB cached instances.
2017-01-28T07:05:36-0500 [stdout#info] Handling ScriptDB cached instances.
2017-01-28T07:05:36-0500 [stdout#info] Syncing sessions.
2017-01-28T07:05:36-0500 [AMPProtocol,0,127.0.0.1] [..] Logged out: Lazarus 72.238.160.155 (0 sessions(s) remaining)
2017-01-28T07:05:43-0500 [AMPProtocol,0,127.0.0.1] [EE] AMP Error for AdminServer2Portal: 2
2017-01-28T07:05:43-0500 [AMPProtocol,0,127.0.0.1] [EE] AMP Error for AdminServer2Portal: 'int' object has no attribute 'disconnect'
2017-01-28T07:05:43-0500 [stdout#info] Calling server reload hook.
2017-01-28T07:05:43-0500 [stdout#info] Saving MONITOR_HANDLER.
2017-01-28T07:05:43-0500 [stdout#info] Saving TICKER_HANDLER.
2017-01-28T07:05:43-0500 [stdout#info] Calling server_stop hook.
2017-01-28T07:05:43-0500 [stdout#info] Before reactor.callLater, 3 second delay
2017-01-28T07:05:43-0500 [stdout#info] End of shutdown method.
2017-01-28T07:05:46-0500 [stdout#info] Beginning shutdown.
2017-01-28T07:05:46-0500 [stdout#info] Inside deferred, returnValue(None)
```
Alas, I have tried repeatedly to quit a secondary player in the midst of reload (both by calling `@quit` and by simply abruptly closing the client window) without being able to see the lockup. 
As an update, this has become much more rare ever since I started really optimizing a lot of the database queries that seemed responsible for the database lock errors I ran into - it's been a few weeks since my last freeze. So it's possible that either it was related to that, or some other seemingly unrelated change may have fixed this in whole or in part.
Although this hasn't happened in about a month on my live server ever since I did all the optimizations in the webpage, I had this happen on my test server earlier today, where I run the django debug toolbar. I was starting to load the webpage, which takes quite a while with the django debug toolbar active, and then remembered that I needed to reload some changes I made. So I started a reload, and it then had the same server hang. So it may actually be tied to some race condition in the webserver trying to load a page and the reload taking place during that - it'd explain why the problem vanished after I optimized different webpages and greatly reduced their loading times.
This is able to be reproduced, finally. For reproduction steps, just have the webpage doing something that will take a long time to load - perhaps have the main view for the index do some expensive/lengthy calculation. While the webpage is loading, try to do a `@reload`. The server will freeze, and does so at the spot where I had placed logging statements before. Now as to fixing it, I have no idea. Is there a way to force the webserver to end any current activity before the reload starts, such as stop views that are in the process of being called?
Happened again after about a month and a half of not seeing it. The freeze occurred when I was reloading while trying to address bad sessions/recursion error in a staff member, as seen in issue #1221 . So it's possible that both errors may share some root cause.
@TehomCD Yes, I can reproduce this by making a test view to a new page and put a `time.sleep(20)` in there. It freezes the reload and the server has to be killed with -9 just as you say in your original post. That is at least reproducible then which is a good start. I'll see what I can find out. 
Double-checking my log, I also saw that the only database lock error I've seen over the entire period happened on the same day as well. So tentatively all of these errors do seem related in some way - at least they all seem to occur around the same time.
I was experimenting with this a little bit to try to find the exact time where the freeze happens. If you put in a `self.services.removeService` call to remove the webserver at the start of shutdown while the webserver is busy, that will cause a hang immediately. Otherwise, the hang is exactly when the callLater of reactor.stop occurs, and is timing dependent: if you have a 30 second long callLater, you can continue to look at webpages while the rest of the shutdown method is waiting on the callLater, and the freeze will only happen if you happen to make another request right when the callLater would land. Since the portal is always okay, and it's only the server that needs to be killed, I wonder if the solution would be some sort of flag in the webserver that says it's alright to stop based on handling requests, and waiting to stop the reactor until after the webserver has stopped? Or maybe there might be a way to kill requests that are currently being handled, so that it's safe to stop it? I'll try to poke more at it later.

Edit: More logging showed that specifically, it's `self.pool.stop()` inside `WSGIWebServer`'s stopService method that's causing the hang, it looks like.
Did some experimentation with creating a separate service just for the Threadpool and setting Evennia's services as its parent, in case for some reason the Webserver shouldn't have had responsibility for handling its shutdown. No luck there - the webserver will shut down fine independently when stopService is called by `reactor.stop()`, but still get the hang when `self.pool.stop()` is called inside the new `ThreadPoolService` object. The nature of this error makes me think that there should be a solution to it, though. Stopping a threadpool safely when it's in use doesn't seem like that weird of a use case to me - surely this would come up for sites that would have much more traffic than mine? 
I don't know what could cause this on Linux. Having reloaded Evennia *thousands* of times, I never encounteted a case when the reloader fails -because the reload mechanism I (as far as I know) very robust under an OS with proper signal handling. One thing I do know is that the reload may take longer or shorter at times, depending on how much cleanup and objets with pre-reload hooks are cached. Only thing I can think of is the reload somehow clashing with the cache flush; might be worth checking if there is a correlation between the flush and the times the reload fails.
@TehomCD Any more data on this with the logging statements we discussed in IRC?
Had it finally happen again after putting in the logging, just now.
```(arx) tehom@arxhost:~/arx/arx$ 2016-12-14T07:07:02-0500 [stdout#info] Beginning shutdown.
2016-12-14T07:07:02-0500 [stdout#info] Setting restart mode.
2016-12-14T07:07:02-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2016-12-14T07:07:02-0500 [stdout#info] Handling ObjectDB cached instances.
2016-12-14T07:07:02-0500 [stdout#info] Handling PlayerDB cached instances.
2016-12-14T07:07:02-0500 [stdout#info] Handling ScriptDB cached instances.
2016-12-14T07:07:03-0500 [stdout#info] Syncing sessions.
2016-12-14T07:07:03-0500 [stdout#info] Calling server reload hook.
2016-12-14T07:07:03-0500 [stdout#info] Saving MONITOR_HANDLER.
2016-12-14T07:07:03-0500 [stdout#info] Beginning shutdown.
```

And that's the end of my print statements, though it looks like I should have added more to be thorough. The last thing seen is before returning the None value in the deferred from the callLater statement. 

After removing the server.pid and trying to start it up again, which got an error about the port already being used, since the twisted process hadn't been killed yet, I got the notification of evennia runner being killed, so it seems to have thought that the evennia_runner was still alive. I then just killed the stuck server process and started it up, and players just saw it as a frighteningly long reload - the portal still always stays alive throughout the process.
@TehomCD So what is missing from that output is just where those strings are printed. So exactly where is "Beginning shutdown" printed?
```
 @defer.inlineCallbacks
    def shutdown(self, mode=None, _reactor_stopping=False):
        """
        Shuts down the server from inside it.

        mode - sets the server restart mode.
               'reload' - server restarts, no "persistent" scripts
                          are stopped, at_reload hooks called.
               'reset' - server restarts, non-persistent scripts stopped,
                         at_shutdown hooks called but sessions will not
                         be disconnected.
               'shutdown' - like reset, but server will not auto-restart.
               None - keep currently set flag from flag file.
        _reactor_stopping - this is set if server is stopped by a kill
                            command OR this method was already called
                             once - in both cases the reactor is
                             dead/stopping already.
        """
        print("Beginning shutdown.")
        if _reactor_stopping and hasattr(self, "shutdown_complete"):
            # this means we have already passed through this method
            # once; we don't need to run the shutdown procedure again.
            defer.returnValue(None)
        print("Setting restart mode.")
        mode = self.set_restart_mode(mode)

        from evennia.objects.models import ObjectDB
        #from evennia.players.models import PlayerDB
        from evennia.server.models import ServerConfig

        if mode == 'reload':
            # call restart hooks
            print("Mode is reload. Setting ServerConfig.objects.conf")
            ServerConfig.objects.conf("server_restart_mode", "reload")
            print("Handling ObjectDB cached instances.")
            yield [o.at_server_reload() for o in ObjectDB.get_all_cached_instances()]
            print("Handling PlayerDB cached instances.")
            yield [p.at_server_reload() for p in PlayerDB.get_all_cached_instances()]
            print("Handling ScriptDB cached instances.")
            yield [(s.pause(manual_pause=False), s.at_server_reload()) for s in ScriptDB.get_all_cached_instances() if s.is_active]
            print("Syncing sessions.")
            yield self.sessions.all_sessions_portal_sync()
            print("Calling server reload hook.")
            self.at_server_reload_stop()
            # only save monitor state on reload, not on shutdown/reset
            from evennia.scripts.monitorhandler import MONITOR_HANDLER
            print("Saving MONITOR_HANDLER.")
            MONITOR_HANDLER.save()
```

It looks like I omitted the first line of the log, 'Beginning shutdown' was also printed at the start. It appears at the start of the method, so the second one, which made the last print statement, seems to be from the callback.
Hm, so we are missing output tests after the monitorhandler save then, from https://github.com/evennia/evennia/blob/master/evennia/server/server.py#L380 and onward. The second call of "Beginning shutdown" is also OK, this is a catch-all because the reactor will react to the shut-down signal on its own). 
But this at least tells us that the calling of all on-object hooks seem to have been finished correctly. What remains are only the tickerhandler resync and the actual stopping of the reactor which is not supposed to be any problem... 

From what you write the evennia runner is not dead and in fact the Server's ports are not even closeing which rules out some sort of problem with the runner going out of scope or signals not passing correctly. This seems to be due to the Twisted reactor not correctly shutting down when sent an external signal but locking up and staying in some sort of limbo. 

It would be good if you could put prints in the small remainder of the `shutdown` method just on the off-chance we can spot where the lock-up is. Especially at the tickerhandler and before/after the actual dcall to stop the reactor would be useful. 
Actually, I would suspect the reactor.stop() actually works too, since this is what causes Twisted to send the responds to and is causing that second call of `shutdown` (the one causing the second "Beginning shutdown"). That signal is setup [here](https://github.com/evennia/evennia/blob/master/evennia/server/server.py#L171).  Since there is no more output after that, I suspect the `shutdown` method goes into the if statement. In which case we won't be able to pin-point it since it then goes into `defer.returnValue(None)`.
I added the print statements to the rest of the shutdown method, so hopefully next time we'll be able to ensure the rest of that finishes correctly. If there's anything else you think that might help to have prints at, let me know and I'll add 'em.
I wonder if `defer.returnValue` would lock up if the reactor has already managed to shut down before it is called. If so it might explain why it happens with irregularity - most often it gets called while the reactor is still in the process of shutting down (just how fast this happens depends on the speed of the signal processing) but some times the reactor is already gone. Not sure.  
Happened again, and the last statement to execute was my print statement before returning the deferred, so it's possible that may be the culprit for some reason.
```
2016-12-18T19:55:37-0500 [stdout#info] Beginning shutdown.
2016-12-18T19:55:37-0500 [stdout#info] Setting restart mode.
2016-12-18T19:55:37-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2016-12-18T19:55:37-0500 [stdout#info] Handling ObjectDB cached instances.
2016-12-18T19:55:37-0500 [stdout#info] Handling PlayerDB cached instances.
2016-12-18T19:55:37-0500 [stdout#info] Handling ScriptDB cached instances.
2016-12-18T19:55:37-0500 [stdout#info] Syncing sessions.
2016-12-18T19:55:37-0500 [stdout#info] Calling server reload hook.
2016-12-18T19:55:37-0500 [stdout#info] Saving MONITOR_HANDLER.
2016-12-18T19:55:37-0500 [stdout#info] Saving TICKER_HANDLER.
2016-12-18T19:55:37-0500 [stdout#info] Calling server_stop hook.
2016-12-18T19:55:37-0500 [stdout#info] Before reactor.callLater
2016-12-18T19:55:37-0500 [stdout#info] End of shutdown method.
2016-12-18T19:55:37-0500 [stdout#info] Beginning shutdown.
2016-12-18T19:55:37-0500 [stdout#info] Inside deferred, returnValue(None)
```

One question - what makes the portal print out '...' while waiting for the server to restart? I noticed it doesn't do that until I kill the hung server process, which then prints out my notification that the evennia runner has been killed. So wherever in the shutdown/restart process it hangs, it stops before the point where it starts printing out ellipses.
The problem seems to have become more frequent with time or game activity, so I'm trying to think of anything related to that - had it hang two times in quick succession recently. I wonder if logging could somehow be to blame? My portal and http logs tend to rotate a few times a day, and when I forget to clean them out for a few days I start having things like `portal.log.27` or whatever. I'll try to diligently remove them on a daily basis and see if the problem goes back to being much more rare.
As a contrast for clarity, here's what a normal reload has that it's missing from the point it hangs:
```
2017-01-07T00:49:12-0500 [-] (TCP Port 5001 Closed)
2017-01-07T00:49:12-0500 [-] (TCP Port 5000 Closed)
2017-01-07T00:49:12-0500 [-] Main loop terminated.
2017-01-07T00:49:12-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.
Server restarting ...
--------------------------------------------------
 Arx Server (0.6.0 (rev 5b06107)) started.
  amp (to Portal): 5000
  webserver: 5001
--------------------------------------------------
2017-01-07T00:49:13-0500 [-] Loading /home/tehom/arx/evennia/evennia/server/server.py...
2017-01-07T00:49:14-0500 [-] Loaded.
2017-01-07T00:49:14-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 16.4.0 (/home/tehom/arx/bin/python2 2.7.12) starting up.
2017-01-07T00:49:14-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2017-01-07T00:49:14-0500 [-] AmpServerFactory starting on 5000
2017-01-07T00:49:14-0500 [-] Website starting on 5001
2017-01-07T00:49:15-0500 [evennia.server.amp.AmpServerFactory] AMPProtocol connection established (HOST:IPv4Address(TCP, '127.0.0.1', 5000) PEER:IPv4Address(TCP, '127.0.0.1', 45424))
```
The ports remain in use until the server process is killed, at which point the evennia runner is also killed as well. Until that point, the Evennia Runner is still running, but doesn't break out of its reload loop: it hasn't yet been sent the signal to restart the server.

I think I'll try removing the `defer.returnValue(None)` statement entirely, just having it do an if/else check instead, and see if that changes anything.
No luck - still hung without with the `defer.returnValue(None)` statement: that block now only prints out the statement about it being inside the deferred block, and does nothing else. Here's the most recent log (snipped messages about staticfiles for brevity), including what happens after I kill the process and start it back up again.
```
2017-01-08T19:22:53-0500 [stdout#info] Beginning shutdown.
2017-01-08T19:22:53-0500 [stdout#info] Setting restart mode.
2017-01-08T19:22:53-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2017-01-08T19:22:53-0500 [stdout#info] Handling ObjectDB cached instances.
2017-01-08T19:22:53-0500 [stdout#info] Handling PlayerDB cached instances.
2017-01-08T19:22:53-0500 [stdout#info] Handling ScriptDB cached instances.
2017-01-08T19:22:54-0500 [stdout#info] Syncing sessions.
2017-01-08T19:22:54-0500 [stdout#info] Calling server reload hook.
2017-01-08T19:22:54-0500 [stdout#info] Saving MONITOR_HANDLER.
2017-01-08T19:22:54-0500 [stdout#info] Saving TICKER_HANDLER.
2017-01-08T19:22:54-0500 [stdout#info] Calling server_stop hook.
2017-01-08T19:22:54-0500 [stdout#info] Before reactor.callLater
2017-01-08T19:22:54-0500 [stdout#info] End of shutdown method.
2017-01-08T19:22:54-0500 [stdout#info] Beginning shutdown.
2017-01-08T19:22:54-0500 [stdout#info] Inside deferred, returnValue(None)

(arx) tehom@arxhost:~/arx/arx$ evennia start

Evennia Server is already running as process 19023. Not restarted.

Evennia Portal is already running as process 7674. Not restarted.
(arx) tehom@arxhost:~/arx/arx$ kill -9 19023
(arx) tehom@arxhost:~/arx/arx$ message is server_stopped, int(rc) is -9, get_restart_mode is reload: 
Evennia runner has been killed!

(arx) tehom@arxhost:~/arx/arx$ cd server
(arx) tehom@arxhost:~/arx/arx/server$ rm server.pid
(arx) tehom@arxhost:~/arx/arx/server$ cd ..
(arx) tehom@arxhost:~/arx/arx$ evennia start

Starting Evennia Server (output to stdout).

Evennia Portal is already running as process 7674. Not restarted.
(arx) tehom@arxhost:~/arx/arx$ --------------------------------------------------
 Arx Server (0.6.0 (rev 6de75ea)) started.
  amp (to Portal): 5000
  webserver: 5001
--------------------------------------------------
2017-01-08T19:23:26-0500 [-] Loading /home/tehom/arx/evennia/evennia/server/server.py...
2017-01-08T19:23:27-0500 [-] Loaded.
2017-01-08T19:23:27-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 16.4.0 (/home/tehom/arx/bin/python2 2.7.12) starting up.
2017-01-08T19:23:27-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2017-01-08T19:23:27-0500 [-] AmpServerFactory starting on 5000
2017-01-08T19:23:27-0500 [-] Website starting on 5001
2017-01-08T19:23:28-0500 [evennia.server.amp.AmpServerFactory] AMPProtocol connection established (HOST:IPv4Address(TCP, '127.0.0.1', 5000) PEER:IPv4Address(TCP, '127.0.0.1', 32802))
```
Going to try changing the callLater for reactor.stop() to be a 1 second delay rather than 0, in case there might be any weird timing issues at play.
@TehomCD It's very strange that this keeps reappearing. If you monitor the processes in a separate window (say with `htop`, filtering on the twistd processes to only show them), could you maybe see if you see any differences in their behavior between a normal reload and one where it fails?
Good idea, I'll start taking a look at that. I haven't had a reload fail in the last few days since setting `callLater` to 1 rather than 0, but who knows if that's just a coincidence: I've gone weeks between a hang before.
Last night, I had what appeared to be a server hang during normal operation, but I have no idea what caused it - it wasn't prompted by an attempt to reload the server or anything else initiated on my part. After a few minutes, I tried to do `evennia reload` from the command line to see if it'd resolve it, and it froze during the attempt to reload. I have no idea if this is related to the hangs during `@reload` that I've seen previously, but I figured I might as well post where it stopped at:

```
Evennia: Server reload.
2017-01-18T11:23:48-0500 [AMPProtocol,0,127.0.0.1] Received SIGINT, shutting down.
(arx) tehom@arxhost:~/arx/arx$ 2017-01-18T11:23:53-0500 [stdout#info] Beginning shutdown.
2017-01-18T11:23:53-0500 [stdout#info] Setting restart mode.
2017-01-18T11:23:53-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2017-01-18T11:23:53-0500 [stdout#info] Handling ObjectDB cached instances.
2017-01-18T11:23:53-0500 [stdout#info] Handling PlayerDB cached instances.
2017-01-18T11:23:53-0500 [stdout#info] Handling ScriptDB cached instances.
2017-01-18T11:23:54-0500 [stdout#info] Syncing sessions.
```
So during this particular freeze, it happened at syncing sessions/before calling the server reload hook, in `yield self.sessions.all_sessions_portal_sync()`, calling either the AMP bridge or the portal process. I had to kill the portal process this time to get it unstuck, while the normal hangs I saw during a `@reload` required killing the server process. So probably something entirely different, but possibly related? Not sure.
Had another of the previous hangs with `callLater(1, reactor.stop)`. It did seem to become more infrequent with 1, but that might just be confirmation bias. I'm going to change it to `callLater(3, reactor.stop)` and see if there's any difference.
If you didn't see it, I should point out that the reload/startup hook's have changed last week, notably the calls of saving the monitor/tickerhandler in evennia/server/server.py. Just in case you need to update your printouts.
As for the problem itself; I wonder if this could be some sort of AMP issue at its core; if you say it freezes without a reload (assuming the problems are related) but the server is still running, there are only a few possible causes:

- AMP freezes or gets clogged up for some reason. This should be noticeable in the portal log. How long ago did you restart the Portal? Many upstream changes require Portal restart (just to rule that out).
- A network or database lockup. If the database slows down dramatically for some reason (e.g. due to filesystem errors) this *will* freeze Evennia, especially during reload. You should catch this with your server printouts though.
- An actual freeze or bug in the server that is swallowed by try...catch events and thus never reach the log. There are no known such catches in the core but it's not impossible. There could also be such in Arx' custom code I guess.

Yeah, I did merge those changes in and adjust my version of `server.py` with the name of the hook changing and so on. Last time I restarted the portal was when I had it freeze on me about two days ago. It's the first time I've seen the portal itself freeze that I can recall, and the behavior was markedly different than the crashes on a `reload` - just several minutes of the game failing to respond to commands, though I saw someone still able to log in, so the portal may have possibly still been working, then the portal hanging upon trying to `evennia reload` from the command line, ending in me having to kill the process.  That particular crash did seem to be AMP or portal related, since it froze right before `yield self.sessions.all_sessions_portal_sync()`, https://github.com/evennia/evennia/blob/master/evennia/server/server.py#L355 . This could have been an example of a database slowdown or network lockup crashing evennia during a reload, maybe?

Other than that, I haven't restarted the portal very regularly - I think it may have been a week or two before that since I last restarted it. Looking at the log, there's the occasional mysterious AMP errors that don't give any information, just an integer, but I kind of doubt they're related - I see those on my test machine too, and I've never had a freeze there.
While trying to investigate the lag issue I have with the latest commit, a player disconnected during a reload process and a server hang immediately followed. Leaving the log here as a possible cause:

```
2017-01-28T07:05:36-0500 [stdout#info] Beginning shutdown.
2017-01-28T07:05:36-0500 [stdout#info] Setting restart mode.
2017-01-28T07:05:36-0500 [stdout#info] Mode is reload. Setting ServerConfig.objects.conf
2017-01-28T07:05:36-0500 [stdout#info] Handling ObjectDB cached instances.
2017-01-28T07:05:36-0500 [stdout#info] Handling PlayerDB cached instances.
2017-01-28T07:05:36-0500 [stdout#info] Handling ScriptDB cached instances.
2017-01-28T07:05:36-0500 [stdout#info] Syncing sessions.
2017-01-28T07:05:36-0500 [AMPProtocol,0,127.0.0.1] [..] Logged out: Lazarus 72.238.160.155 (0 sessions(s) remaining)
2017-01-28T07:05:43-0500 [AMPProtocol,0,127.0.0.1] [EE] AMP Error for AdminServer2Portal: 2
2017-01-28T07:05:43-0500 [AMPProtocol,0,127.0.0.1] [EE] AMP Error for AdminServer2Portal: 'int' object has no attribute 'disconnect'
2017-01-28T07:05:43-0500 [stdout#info] Calling server reload hook.
2017-01-28T07:05:43-0500 [stdout#info] Saving MONITOR_HANDLER.
2017-01-28T07:05:43-0500 [stdout#info] Saving TICKER_HANDLER.
2017-01-28T07:05:43-0500 [stdout#info] Calling server_stop hook.
2017-01-28T07:05:43-0500 [stdout#info] Before reactor.callLater, 3 second delay
2017-01-28T07:05:43-0500 [stdout#info] End of shutdown method.
2017-01-28T07:05:46-0500 [stdout#info] Beginning shutdown.
2017-01-28T07:05:46-0500 [stdout#info] Inside deferred, returnValue(None)
```
Alas, I have tried repeatedly to quit a secondary player in the midst of reload (both by calling `@quit` and by simply abruptly closing the client window) without being able to see the lockup. 
As an update, this has become much more rare ever since I started really optimizing a lot of the database queries that seemed responsible for the database lock errors I ran into - it's been a few weeks since my last freeze. So it's possible that either it was related to that, or some other seemingly unrelated change may have fixed this in whole or in part.
Although this hasn't happened in about a month on my live server ever since I did all the optimizations in the webpage, I had this happen on my test server earlier today, where I run the django debug toolbar. I was starting to load the webpage, which takes quite a while with the django debug toolbar active, and then remembered that I needed to reload some changes I made. So I started a reload, and it then had the same server hang. So it may actually be tied to some race condition in the webserver trying to load a page and the reload taking place during that - it'd explain why the problem vanished after I optimized different webpages and greatly reduced their loading times.
This is able to be reproduced, finally. For reproduction steps, just have the webpage doing something that will take a long time to load - perhaps have the main view for the index do some expensive/lengthy calculation. While the webpage is loading, try to do a `@reload`. The server will freeze, and does so at the spot where I had placed logging statements before. Now as to fixing it, I have no idea. Is there a way to force the webserver to end any current activity before the reload starts, such as stop views that are in the process of being called?
Happened again after about a month and a half of not seeing it. The freeze occurred when I was reloading while trying to address bad sessions/recursion error in a staff member, as seen in issue #1221 . So it's possible that both errors may share some root cause.
@TehomCD Yes, I can reproduce this by making a test view to a new page and put a `time.sleep(20)` in there. It freezes the reload and the server has to be killed with -9 just as you say in your original post. That is at least reproducible then which is a good start. I'll see what I can find out. 
Double-checking my log, I also saw that the only database lock error I've seen over the entire period happened on the same day as well. So tentatively all of these errors do seem related in some way - at least they all seem to occur around the same time.
I was experimenting with this a little bit to try to find the exact time where the freeze happens. If you put in a `self.services.removeService` call to remove the webserver at the start of shutdown while the webserver is busy, that will cause a hang immediately. Otherwise, the hang is exactly when the callLater of reactor.stop occurs, and is timing dependent: if you have a 30 second long callLater, you can continue to look at webpages while the rest of the shutdown method is waiting on the callLater, and the freeze will only happen if you happen to make another request right when the callLater would land. Since the portal is always okay, and it's only the server that needs to be killed, I wonder if the solution would be some sort of flag in the webserver that says it's alright to stop based on handling requests, and waiting to stop the reactor until after the webserver has stopped? Or maybe there might be a way to kill requests that are currently being handled, so that it's safe to stop it? I'll try to poke more at it later.

Edit: More logging showed that specifically, it's `self.pool.stop()` inside `WSGIWebServer`'s stopService method that's causing the hang, it looks like.
Did some experimentation with creating a separate service just for the Threadpool and setting Evennia's services as its parent, in case for some reason the Webserver shouldn't have had responsibility for handling its shutdown. No luck there - the webserver will shut down fine independently when stopService is called by `reactor.stop()`, but still get the hang when `self.pool.stop()` is called inside the new `ThreadPoolService` object. The nature of this error makes me think that there should be a solution to it, though. Stopping a threadpool safely when it's in use doesn't seem like that weird of a use case to me - surely this would come up for sites that would have much more traffic than mine? 

Resolving patch:
diff --git a/evennia/server/server.py b/evennia/server/server.py
--- a/evennia/server/server.py
+++ b/evennia/server/server.py
@@ -151,7 +151,8 @@ def __init__(self, application):
         sys.path.insert(1, '.')
 
         # create a store of services
-        self.services = service.IServiceCollection(application)
+        self.services = service.MultiService()
+        self.services.setServiceParent(application)
         self.amp_protocol = None  # set by amp factory
         self.sessions = SESSIONS
         self.sessions.server = self
@@ -348,6 +349,10 @@ def shutdown(self, mode=None, _reactor_stopping=False):
         from evennia.server.models import ServerConfig
         from evennia.utils import gametime as _GAMETIME_MODULE
 
+        if WEBSERVER_ENABLED:
+            # finish all pending web requests. Otherwise stopping threadpool will cause deadlock.
+            yield self.web_root.get_pending_requests()
+
         if mode == 'reload':
             # call restart hooks
             ServerConfig.objects.conf("server_restart_mode", "reload")
@@ -392,8 +397,15 @@ def shutdown(self, mode=None, _reactor_stopping=False):
             # this will also send a reactor.stop signal, so we set a
             # flag to avoid loops.
             self.shutdown_complete = True
-            # kill the server
-            reactor.callLater(1, reactor.stop)
+            if WEBSERVER_ENABLED:
+                # Just to be extra sure, get all pending requests that might have occurred after we started
+                d = self.web_root.get_pending_requests()
+                d.addCallback(lambda _: reactor.stop())
+                from twisted.internet import task
+                yield task.deferLater(reactor, 1, d.callback, None)
+            else:
+                # kill the server
+                reactor.callLater(1, reactor.stop)
 
         # we make sure the proper gametime is saved as late as possible
         ServerConfig.objects.conf("runtime", _GAMETIME_MODULE.runtime())
@@ -533,11 +545,13 @@ def at_server_cold_stop(self):
     # recognized by Django
     threads = threadpool.ThreadPool(minthreads=max(1, settings.WEBSERVER_THREADPOOL_LIMITS[0]),
                                     maxthreads=max(1, settings.WEBSERVER_THREADPOOL_LIMITS[1]))
+
     web_root = DjangoWebRoot(threads)
     # point our media resources to url /media
     web_root.putChild("media", static.File(settings.MEDIA_ROOT))
     # point our static resources to url /static
     web_root.putChild("static", static.File(settings.STATIC_ROOT))
+    EVENNIA.web_root = web_root
 
     if WEB_PLUGINS_MODULE:
         # custom overloads
diff --git a/evennia/server/webserver.py b/evennia/server/webserver.py
--- a/evennia/server/webserver.py
+++ b/evennia/server/webserver.py
@@ -28,6 +28,7 @@
 _UPSTREAM_IPS = settings.UPSTREAM_IPS
 _DEBUG = settings.DEBUG
 
+
 #
 # X-Forwarded-For Handler
 #
@@ -117,6 +118,8 @@ class DjangoWebRoot(resource.Resource):
     understands by tweaking the way
     child instancee ars recognized.
     """
+    open_requests = []
+
     def __init__(self, pool):
         """
         Setup the django+twisted resource.
@@ -128,6 +131,21 @@ def __init__(self, pool):
         resource.Resource.__init__(self)
         self.wsgi_resource = WSGIResource(reactor, pool, WSGIHandler())
 
+    def get_pending_requests(self):
+        """
+        Converts our open_requests list of deferreds into a DeferredList
+
+        Returns:
+            d_list (deferred): A DeferredList object of all our requests
+        """
+        from twisted.internet import defer
+        return defer.DeferredList(self.open_requests, consumeErrors=True)
+
+    def _decrement_requests(self, *args, **kwargs):
+        deferred = kwargs.get('deferred', None)
+        if deferred in self.open_requests:
+            self.open_requests.remove(deferred)
+
     def getChild(self, path, request):
         """
         To make things work we nudge the url tree to make this the
@@ -140,8 +158,12 @@ def getChild(self, path, request):
         """
         path0 = request.prepath.pop(0)
         request.postpath.insert(0, path0)
+        deferred = request.notifyFinish()
+        self.open_requests.append(deferred)
+        deferred.addBoth(self._decrement_requests, deferred=deferred)
         return self.wsgi_resource
 
+
 #
 # Site with deactivateable logging
 #
@@ -151,11 +173,13 @@ class Website(server.Site):
     This class will only log http requests if settings.DEBUG is True.
     """
     noisy = False
+
     def log(self, request):
-        "Conditional logging"
+        """Conditional logging"""
         if _DEBUG:
             server.Site.log(self, request)
 
+
 #
 # Threaded Webserver
 #


Test patch: