Repository: learningequality/kolibri

Issue:
running kolibri pex on mac in background mode causes a seg fault

### Observed behavior

Trying to run the kolibri pex in background mode causes a hard python crash with no script-level errors:

![image](https://user-images.githubusercontent.com/2367265/51886486-e815c580-2344-11e9-8227-4576e57214f1.png)

running with `--foreground` does not have this issue

### Expected behavior

no crash

### User-facing consequences

frustration

### Errors and logs

This is the top of the report

```
Crashed Thread:        2

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x000000011207bb36
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [54294]
```

I can supply additional info

### Steps to reproduce

```kolibri start```


### Context

0.12



Discussion:
This problem seems to only exist for python2.
When running the pex with python3, server starts successfully.
Appears to be the call to https://github.com/learningequality/kolibri/blob/develop/kolibri/utils/cli.py#L322
https://github.com/learningequality/kolibri/blob/develop/kolibri/utils/system.py#L185
https://github.com/learningequality/kolibri/blob/develop/kolibri/utils/system.py#L79
which creates the segfault cc: @benjaoming 
Good catch, @ralphiee22! This change looks likely to be the culprit: 

https://github.com/learningequality/kolibri/commit/16b98b6c9cee1cac47bc2def2fb9ff42e1e1a902#diff-321c4ae9ad388f68b2ca687745d0b43f

I wonder if we need to keep the hardcoding of `daemon=False` under Python 2.7, or if we need to apply additional fixes there? @rtibbles Any ideas on why this change may not work on 2.7?
@ralphiee22 and I had a look through this together in person, and we think @kollivier's idea is the most practical solution.

There seem to be multiple reports of issues with Python and MacOS forking issues, but I can't for the life of me figure out why it is Python 2 only, and only in the pex file.
Hmm, was googling and I came across this, which basically states that High Sierra was changed to improve security by immediately throwing an error when a forked process attempts to call a disallowed function:

https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/

Are these crashes happening under High Sierra? I wonder if this is a case where internal Python 2.7 code (maybe OS X only, but possibly not) isn't handling forking properly, and High Sierra is now throwing an explicit error rather than continuing as previous versions did.  Looks like we can test if this is the issue by setting the following environment variable: 

`export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES`
@ralphiee22 tried running with that env var, and it didn't seem to fix the issue :/
> Are these crashes happening under High Sierra?

Mojave - MacOS 10.14.2 (18C54)
How big of an issue would it be if we were to re-introduce the `daemon=False` hardcoded value on Mac for Python 2.7? The fact that it's hitting 2.7 and not 3.x makes me think this is one of those cases where an issue was found and fixed in 3.x but they never backported the fix. I don't know if we can quickly get to the bottom of this, though, so maybe restoring the old behavior on 2.7 for Mac is the short term fix.
Is there a way to be Python3-only on Mac?
> Is there a way to be Python3-only on Mac?

Mac system python is 2.7
> maybe restoring the old behavior on 2.7 for Mac is the short term fix.

that sounds fine to me
Actually I'm still encountering issues with this:

```
objc[52180]: +[NSValue initialize] may have been in progress in another thread when fork() was called.
objc[52180]: +[NSValue initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
```

It starts but that quickly crashes
I haven't seen this in a while - closing
I'm seeing it in beta 12, trying to follow the reproduction steps here https://github.com/learningequality/kolibri/issues/5112#issuecomment-468763062

I get these console messages, then the mac Crash pop up shows up

```
WARNING  >>>>>> AsyncNotificationsThread.start_command: 4567573952
INFO     Initializing background log saving process
WARNING  >>>>>> AsyncNotificationsThread.run: 123145414094848
WARNING  >>>>>> AsyncNotificationQueue.start: 123145414094848
WARNING  		 len(self.running): []
WARNING  		 len(self.queue): []
```
yeah - confirmed
Can I ask why we are using fork here and why `--foreground` is not the default? There are ways of making processes run as daemons on pretty much every OS, so I'm wondering why we don't take advantage of those.

When a process forks, the child process does not inherit the parent process's threads, so in practice  calling fork does not always result in an exact clone of the process, and can cause some nasty problems. See this SO post for example: 

https://stackoverflow.com/questions/12820248/what-happens-to-running-threads-after-forking

Note that we not only have to worry about the code that we write, but also what any libraries we've called / are using are doing. I think if there's a way to solve this problem without forking, we should do it. If not, then I think we should make the default not to fork and fork right after the process starts, even before we start initializing anything.
`--foreground` is not the default, because we started from the perspective of using the pex file as the default install method. I think you are right that we should try to use more OS specific daemonization methods from within our installers, to avoid these issues arising (but we don't have a macOS installer yet!)

Also, a recent issue with the notifications logging thread succumbed to the exact conditions described in the SO post, so this is a very real concern.
What if we just created a `kolibri register-service` command which did the OS-specific steps to register with the system service manager as a daemon? (would probably require sudo / admin of course) I think then we could do it all from the command line without having to go through the trouble of creating an installer. 
Our strategy so far has been to do that inside installers because it has been simpler not to do the work from python land, and because the steps are normally sufficiently distinct, having them in the common repo has seemed to be more bother (incrementing Kolibri versions to support platform specific changes). 
Okay, so what do you think is the best way forward here? For the upcoming release, I feel like maybe a good first start is to print out a warning on the command line when running in the background saying this approach could cause issues, with a link to docs on how to register Kolibri as a system service on your platform of choice.

That would leave the installers to simplify things. On the Mac side, it took me about 15 minutes to tweak the kolibri-installer-android sources to spit out a "Kolibri.app" that fires up and runs Kolibri, so that is one avenue we could pursue. 
The best way forward is probably to revert the background default for Mac until we can do a proper app wrapper that does the daemonization use MacOS specific tools.

As MacOS as a production target is low priority, I think this is the simplest way forward for now.
Also worth noting that I think this behaviour is particularly unique to MacOS because of the way it handles forked processes:
> You must not daemonize your process. This includes calling the daemon function, calling fork followed by exec, or calling fork followed by exit. If you do, launchd thinks your process has died. Depending on your property list key settings, launchd will either keep trying to relaunch your process until it gives up (with a “respawning too fast” error message) or will be unable to restart it if it really does die.

From https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html

Obviously this is from the MacOS daemonization docs, but I assume that similar monitoring of processes on MacOS could be causing issues there as well - basically deallocating memory from a forked process, where the parent has called exit.

Resolving patch:
diff --git a/kolibri/utils/cli.py b/kolibri/utils/cli.py
--- a/kolibri/utils/cli.py
+++ b/kolibri/utils/cli.py
@@ -668,6 +668,8 @@ def main(args=None):  # noqa: max-complexity=13
         call_command("clearsessions")
 
         daemon = not arguments['--foreground']
+        if sys.platform == 'darwin':
+            daemon = False
         start(port, daemon=daemon)
         return
 


Test patch: