Repository: voxel51/fiftyone

Issue:
Document that the FiftyOne App can't be opened from a python script without `session.wait()`
We had a user run this snippet from the docs

```py
import fiftyone as fo
import fiftyone.zoo as foz
import fiftyone.brain as fob

# Load the CIFAR-10 test split
# Downloads the dataset from the web if necessary
dataset = foz.load_zoo_dataset("cifar10", split="test")

# Compute uniqueness
fob.compute_uniqueness(dataset)

# Sort in increasing order of uniqueness (least unique first)
dups_view = dataset.view().sort_by("uniqueness")

# Launch the App
session = fo.launch_app(view=dups_view)
```

BUT he did this from a Python script, so the app opened and then immediately closed! He thought there was a bug.

The solution is to add `session.wait()` to the end of the script; this needs to be documented somehow so that users that want to use scripts can figure out what to do.

Maybe a clever way of doing this is to print a message from the code itself that recommends `session.wait()` whenever an app is opened and then closed a few milliseconds later?
[FR] Session.wait() equivalent for remote session backend
A user tried creating a remote session in a script, and ran into an issue where the server shut down immediately because the script exited. This is essentially another case of #281, but specific to the server service instead of the app, and with different possible solutions. Specifically, `session.wait()` doesn't work because it only waits for the app to exit, because the server is intended to stay running and allow an app to reconnect later. There are some alternatives that can be used in a script to keep the server alive, like `time.sleep(1e9)` and `signal.sigwait([signal.SIGINT])`, but they aren't as expressive.

The built-in [socketserver](https://docs.python.org/3/library/socketserver.html) class has `handle_request()` and `serve_forever()` methods. We could adopt something similar, noting that exposing them under a `session.server` object instead of directly under `session` might make more sense.

### What areas of FiftyOne does this feature affect?

-   [ ] `App`: FiftyOne application
-   [x] `Core`: Core `fiftyone` Python library
-   [x] `Server`: FiftyOne server



Discussion:
> Maybe a clever way of doing this is to print a message from the code itself that recommends session.wait() whenever an app is opened and then closed a few milliseconds later?

This is an interesting idea. We could probably detect this in _service_main.py, ~~and display a message if the parent process is a non-interactive Python process (e.g. `sys.__stdin__.isatty()`)~~. That check might fail in a notebook, but as long as the app is open long enough, it wouldn't matter.

I think it's also important to mention in the documentation, though. Maybe a note that "These examples/tutorials/recipes are all meant to be run interactively" at the top of pages like https://voxel51.com/docs/fiftyone/tutorials/index.html and https://voxel51.com/docs/fiftyone/recipes/index.html, and/or in the generated notes at the top of notebooks like https://voxel51.com/docs/fiftyone/tutorials/label_mistakes.html
One case I thought of where the approach of logging a message if the app closes quickly won't work: a user could launch the app, then perform some expensive computation (e.g. calculating uniqueness) before the script ends. Also, the `isatty()` check I suggested earlier won't differentiate between an interactive session and a script run from an interactive terminal. The service entrypoint does know if the main process or the child process exited first, so we could use that as an additional check, but I haven't found a way to reliably check for an interactive Python session.


Resolving patch:
diff --git a/fiftyone/core/cli.py b/fiftyone/core/cli.py
--- a/fiftyone/core/cli.py
+++ b/fiftyone/core/cli.py
@@ -727,11 +727,9 @@ def _watch_session(session, remote=False):
     try:
         if remote:
             print("\nTo exit, press ctrl + c\n")
-            while True:
-                time.sleep(60)
         else:
             print("\nTo exit, close the app or press ctrl + c\n")
-            session.wait()
+        session.wait()
     except KeyboardInterrupt:
         pass
 
diff --git a/fiftyone/core/session.py b/fiftyone/core/session.py
--- a/fiftyone/core/session.py
+++ b/fiftyone/core/session.py
@@ -5,8 +5,9 @@
 | `voxel51.com <https://voxel51.com/>`_
 |
 """
-import logging
 from collections import defaultdict
+import logging
+import time
 
 import fiftyone.core.client as foc
 import fiftyone.core.service as fos
@@ -135,6 +136,11 @@ class Session(foc.HasClient):
     def __init__(self, dataset=None, view=None, port=5151, remote=False):
         self._port = port
         self._remote = remote
+        # maintain a reference to prevent garbage collection
+        self._get_time = time.perf_counter
+        self._WAIT_INSTRUCTIONS = _WAIT_INSTRUCTIONS
+        self._disable_wait_warning = False
+
         global _server_services  # pylint: disable=global-statement
         if port not in _server_services:
             _server_services[port] = fos.ServerService(port)
@@ -157,6 +163,7 @@ def __init__(self, dataset=None, view=None, port=5151, remote=False):
                 _REMOTE_INSTRUCTIONS.strip()
                 % (self.server_port, self.server_port, self.server_port)
             )
+        self._start_time = self._get_time()
 
     def __del__(self):
         """Deletes the Session by removing it from the `_subscribed_sessions`
@@ -165,6 +172,13 @@ def __del__(self):
         subscribed.
         """
         try:
+            if (
+                not self._disable_wait_warning
+                and self._get_time() - self._start_time < 2.5
+            ):
+                # logger may already have been garbage-collected
+                print(self._WAIT_INSTRUCTIONS)
+
             global _subscribed_sessions  # pylint: disable=global-statement
             _subscribed_sessions[self._port].discard(self)
 
@@ -264,14 +278,20 @@ def close(self):
         self._update_state()
 
     def wait(self):
-        """Waits for the FiftyOne App to be closed by the user.
+        """Waits for the session to be closed by the user.
 
-        This requires a local (not remote) session.
+        For local sessions, this will wait until the app is closed by the user.
+        For remote sessions, this will wait until the server shuts down, which
+        typically requires interrupting the calling process with Ctrl-C.
         """
-        if self._remote:
-            raise ValueError("Cannot `wait()` for remote sessions to close")
-
-        self._app_service.wait()
+        try:
+            if self._remote:
+                _server_services[self._port].wait()
+            else:
+                self._app_service.wait()
+        except KeyboardInterrupt:
+            self._disable_wait_warning = True
+            raise
 
     # PRIVATE #################################################################
 
@@ -294,3 +314,9 @@ def _update_state(self):
 and then connect to the app on that machine using either
 `fiftyone app connect` or from Python via `fiftyone.launch_app()`.
 """
+
+_WAIT_INSTRUCTIONS = """
+A session appears to have terminated shortly after it was started. If you
+intended to start an app instance or a remote session from a script, you
+should call `session.wait()` to keep the session (and the script) alive.
+"""


Test patch:
diff --git a/tests/session_tests.py b/tests/session_tests.py
new file mode 100644
--- /dev/null
+++ b/tests/session_tests.py
@@ -0,0 +1,49 @@
+"""
+Tests related to Session behavior.
+
+| Copyright 2017-2020, Voxel51, Inc.
+| `voxel51.com <https://voxel51.com/>`_
+|
+"""
+
+import os
+import sys
+import subprocess
+
+import fiftyone as fo
+import fiftyone.core.session as fos
+
+
+def _run_helper(*args):
+    # -u: unbuffered output, to ensure that the warning gets captured
+    return (
+        subprocess.check_output(
+            [
+                sys.executable,
+                "-u",
+                os.path.join(
+                    os.path.dirname(os.path.abspath(__file__)),
+                    "utils",
+                    "session_helper.py",
+                ),
+            ]
+            + list(args)
+        )
+        .decode()
+        .replace("\r", "")
+    )
+
+
+def test_fast_shutdown(capsys):
+    out = _run_helper()
+    assert fos._WAIT_INSTRUCTIONS in out
+
+
+def test_fast_shutdown_remote(capsys):
+    out = _run_helper("--remote")
+    assert fos._WAIT_INSTRUCTIONS in out
+
+
+def test_slow_shutdown(capsys):
+    out = _run_helper("--slow")
+    assert fos._WAIT_INSTRUCTIONS not in out
diff --git a/tests/utils/session_helper.py b/tests/utils/session_helper.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/session_helper.py
@@ -0,0 +1,14 @@
+import argparse
+import sys
+
+import fiftyone as fo
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--remote", action="store_true")
+parser.add_argument("--slow", action="store_true")
+args = parser.parse_args()
+
+session = fo.launch_app(remote=args.remote)
+if args.slow:
+    assert isinstance(session._disable_wait_warning, bool)
+    session._disable_wait_warning = True