Repository: ipython/ipython

Issue:
IPython 5 - Embeded shell not reopening once exited
I recently updated to IPython 5 and am observing some strange behavior with the embedded shell functionality. If ipshell is called more than once in the code, once the first embedded shell is exited, the rest are also exited without allowing for interactivity.

So for example, from the [IPython 5 documentation on embedding](https://ipython.readthedocs.io/en/stable/interactive/reference.html#embedding), the sample code provided in embed_class_long does not work as intended. Once the first instance of ipshell is exited, IPython does not stop again when it is called. Below is an excerpt of the code from the documentation to highlight the issue.

``` python
from __future__ import print_function
from traitlets.config.loader import Config
cfg = Config()
# First import the embeddable shell class
from IPython.terminal.embed import InteractiveShellEmbed

ipshell = InteractiveShellEmbed(config=cfg,
                       banner1 = 'Dropping into IPython',
                       exit_msg = 'Leaving Interpreter, back to program.')

ipshell2 = InteractiveShellEmbed(config=cfg,
                        banner1 = 'Second IPython instance.')

print('\nHello. This is printed from the main controller program.\n')

ipshell('***Called from top level. '
        'Hit Ctrl-D to exit interpreter and continue program.\n'
        'Note that if you use %kill_embedded, you can fully deactivate\n'
        'This embedded instance so it will never turn on again')

print('\nBack in caller program, moving along...\n')

def foo(m):
    s = 'spam'
    ipshell('***In foo(). Try %whos, or print s or m:')
    print('foo says m = ',m)

# Some calls to the above functions which will trigger IPython:
print('Main program calling foo("eggs")\n')
foo('eggs')

ipshell2()
```

On my machine, when this runs, I drop into the first ipshell instance with the "Called from the top level" message. I then type exit to leave this embedded shell. I get the exit message as intended, get the print statement "Back in caller program, moving along..." and the statement "Main program calling foo("eggs")" as intended. Then on my shell, I get the "Dropping into IPython" message which is the banner for the first instance of ipshell, but IPython does not stop. Without embedding a shell, the program continues by printing the  "In foo()" statement, as well as the "Leaving Interpreter" exit message. Finally the script gets to ipshell2() and correctly opens a the second instance. 

Before in IPython 4.x, when foo('eggs') was run, a shell would be embedded. The issue is also a problem when used in for loops. If ipshell is called in a for loop and subsequently exited, all the ipshell calls afterwards will be skipped, which is annoying when you want to stop in every iteration of the for loop to inspect variables.

Here is some system information in case it is relevant.

```
'commit_hash': u'1ba246d',
'commit_source': 'installation',
'default_encoding': 'cp437',
'ipython_path': 'C:\\Anaconda\\lib\\site-packages\\IPython',
'ipython_version': '5.0.0',
'os_name': 'nt',
'platform': 'Windows-7-6.1.7601-SP1',
'sys_executable': 'C:\\Anaconda\\python.exe',
'sys_platform': 'win32',
'sys_version': '2.7.11 |Anaconda custom (64-bit)| (default, Feb 16 2016, 09:58:36) [MSC v.1500 64 bit (AMD64)]'}
```



Discussion:
We changed a bit how it decides when to deactivate an embedded shell, there must be something not quite right in that implementation. Pinging @Carreau.

@Carreau will you have time to look at this soon? Bump the milestone to 5.3 if not.
> @Carreau will you have time to look at this soon? Bump the milestone to 5.3 if not.

I might. I'll try to find some time to review all non-assigned and 5.2 issues soon. 
> I might. I'll try to find some time to review all non-assigned and 5.2 issues soon.

Ok, so first thanks a lot for the detailed explanation and bug report. 

I've just tried on 5.1 and 5.0 both on linux and OS X, and I can't seem to be able to reproduce the bug. 
I'll see if I can get a windows vm tomorrow, and hope that it comes from windows.

```
bash-4.3# python test.py

Hello. This is printed from the main controller program.

Dropping into IPython

***Called from top level. Hit Ctrl-D to exit interpreter and continue program.
Note that if you use %kill_embedded, you can fully deactivate
This embedded instance so it will never turn on again

In [1]: %kill_embedded
Are you sure you want to kill this embedded instance? [y/N]  Y
This embedded IPython will not reactivate anymore once you exit.

In [2]:
Do you really want to exit ([y]/n)?
Leaving Interpreter, back to program.

Back in caller program, moving along...

Main program calling foo("eggs")

foo says m =  eggs
Second IPython instance.
In [1]: %kill_embedded
Are you sure you want to kill this embedded instance? [y/N]  Y
This embedded IPython will not reactivate anymore once you exit.

In [2]:
Do you really want to exit ([y]/n)?

bash-4.3# python -c 'import IPython; print(IPython.sys_info())'
{'commit_hash': u'1ba246d',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/usr/lib/python2.7/site-packages/IPython',
 'ipython_version': '5.0.0',
 'os_name': 'posix',
 'platform': 'Linux-4.4.27-boot2docker-x86_64-with',
 'sys_executable': '/usr/bin/python',
 'sys_platform': 'linux2',
 'sys_version': '2.7.13 (default, Dec 20 2016, 16:33:50) \n[GCC 6.2.1 20160822]'}
```

I'm a bit astonished as how this could be os-specific, but as we have a +1 from @wronglink I'm going to assume you are not the only one.

@wronglink do you experience this on windows as well ?
No , forget what I said. Don't debug at 10pm in the evening.

Resolving patch:
diff --git a/IPython/terminal/embed.py b/IPython/terminal/embed.py
--- a/IPython/terminal/embed.py
+++ b/IPython/terminal/embed.py
@@ -10,6 +10,7 @@
 import warnings
 
 from IPython.core import ultratb, compilerop
+from IPython.core import magic_arguments
 from IPython.core.magic import Magics, magics_class, line_magic
 from IPython.core.interactiveshell import DummyMod, InteractiveShell
 from IPython.terminal.interactiveshell import TerminalInteractiveShell
@@ -25,22 +26,68 @@ class KillEmbeded(Exception):pass
 class EmbeddedMagics(Magics):
 
     @line_magic
+    @magic_arguments.magic_arguments()
+    @magic_arguments.argument('-i', '--instance', action='store_true',
+                              help='Kill instance instead of call location')
+    @magic_arguments.argument('-x', '--exit', action='store_true',
+                              help='Also exit the current session')
+    @magic_arguments.argument('-y', '--yes', action='store_true',
+                              help='Do not ask confirmation')
     def kill_embedded(self, parameter_s=''):
-        """%kill_embedded : deactivate for good the current embedded IPython.
+        """%kill_embedded : deactivate for good the current embedded IPython
 
         This function (after asking for confirmation) sets an internal flag so
-        that an embedded IPython will never activate again.  This is useful to
-        permanently disable a shell that is being called inside a loop: once
-        you've figured out what you needed from it, you may then kill it and
-        the program will then continue to run without the interactive shell
-        interfering again.
+        that an embedded IPython will never activate again for the given call
+        location. This is useful to permanently disable a shell that is being
+        called inside a loop: once you've figured out what you needed from it,
+        you may then kill it and the program will then continue to run without
+        the interactive shell interfering again.
+
+
+        Kill Instance Option
+        --------------------
+
+        If for some reasons you need to kill the location where the instance is
+        created and not called, for example if you create a single instance in
+        one place and debug in many locations, you can use the ``--instance``
+        option to kill this specific instance. Like for the ``call location``
+        killing an "instance" should work even if it is recreated within a
+        loop.
+
+        .. note::
+
+            This was the default behavior before IPython 5.2
+
         """
 
-        kill = ask_yes_no("Are you sure you want to kill this embedded instance? [y/N] ",'n')
-        if kill:
-            self.shell.embedded_active = False
-            print ("This embedded IPython will not reactivate anymore "
-                   "once you exit.")
+        args = magic_arguments.parse_argstring(self.kill_embedded, parameter_s)
+        print(args)
+        if args.instance:
+            # let no ask
+            if not args.yes:
+                kill = ask_yes_no(
+                    "Are you sure you want to kill this embedded instance? [y/N] ", 'n')
+            else:
+                kill = True
+            if kill:
+                self.shell._disable_init_location()
+                print("This embedded IPython instance will not reactivate anymore "
+                      "once you exit.")
+        else:
+            if not args.yes:
+                kill = ask_yes_no(
+                    "Are you sure you want to kill this embedded call_location? [y/N] ", 'n')
+            else:
+                kill = True
+            if kill:
+                self.shell.embedded_active = False
+                print("This embedded IPython  call location will not reactivate anymore "
+                      "once you exit.")
+
+        if args.exit:
+            # Ask-exit does not really ask, it just set internals flags to exit
+            # on next loop.
+            self.shell.ask_exit()
 
 
     @line_magic
@@ -77,29 +124,37 @@ class InteractiveShellEmbed(TerminalInteractiveShell):
 
     @property
     def embedded_active(self):
-        return self._call_location_id not in InteractiveShellEmbed._inactive_locations
+        return (self._call_location_id not in InteractiveShellEmbed._inactive_locations)\
+            and (self._init_location_id not in InteractiveShellEmbed._inactive_locations)
+
+    def _disable_init_location(self):
+        """Disable the current Instance creation location"""
+        InteractiveShellEmbed._inactive_locations.add(self._init_location_id)
 
     @embedded_active.setter
     def embedded_active(self, value):
-        if value :
-            if self._call_location_id in InteractiveShellEmbed._inactive_locations:
-                InteractiveShellEmbed._inactive_locations.remove(self._call_location_id)
+        if value:
+            InteractiveShellEmbed._inactive_locations.discard(
+                self._call_location_id)
+            InteractiveShellEmbed._inactive_locations.discard(
+                self._init_location_id)
         else:
-            InteractiveShellEmbed._inactive_locations.add(self._call_location_id)
+            InteractiveShellEmbed._inactive_locations.add(
+                self._call_location_id)
 
     def __init__(self, **kw):
-        
-    
         if kw.get('user_global_ns', None) is not None:
-            raise DeprecationWarning("Key word argument `user_global_ns` has been replaced by `user_module` since IPython 4.0.")
+            raise DeprecationWarning(
+                "Key word argument `user_global_ns` has been replaced by `user_module` since IPython 4.0.")
 
-        self._call_location_id =  kw.pop('_call_location_id', None)
+        clid = kw.pop('_init_location_id', None)
+        if not clid:
+            frame = sys._getframe(1)
+            clid = '%s:%s' % (frame.f_code.co_filename, frame.f_lineno)
+        self._init_location_id = clid
 
         super(InteractiveShellEmbed,self).__init__(**kw)
 
-        if not self._call_location_id:
-            frame = sys._getframe(1)
-            self._call_location_id = '%s:%s' % (frame.f_code.co_filename, frame.f_lineno)
         # don't use the ipython crash handler so that user exceptions aren't
         # trapped
         sys.excepthook = ultratb.FormattedTB(color_scheme=self.colors,
@@ -107,6 +162,9 @@ def __init__(self, **kw):
                                              call_pdb=self.pdb)
 
     def init_sys_modules(self):
+        """
+        Explicitly overwrite :any:`IPython.core.interactiveshell` to do nothing.
+        """
         pass
 
     def init_magics(self):
@@ -114,7 +172,7 @@ def init_magics(self):
         self.register_magics(EmbeddedMagics)
 
     def __call__(self, header='', local_ns=None, module=None, dummy=None,
-                 stack_depth=1, global_ns=None, compile_flags=None):
+                 stack_depth=1, global_ns=None, compile_flags=None, **kw):
         """Activate the interactive interpreter.
 
         __call__(self,header='',local_ns=None,module=None,dummy=None) -> Start
@@ -131,7 +189,16 @@ def __call__(self, header='', local_ns=None, module=None, dummy=None,
         can still have a specific call work by making it as IPShell(dummy=False).
         """
 
+        # we are called, set the underlying interactiveshell not to exit.
+        self.keep_running = True
+
         # If the user has turned it off, go away
+        clid = kw.pop('_call_location_id', None)
+        if not clid:
+            frame = sys._getframe(1)
+            clid = '%s:%s' % (frame.f_code.co_filename, frame.f_lineno)
+        self._call_location_id = clid
+
         if not self.embedded_active:
             return
 
@@ -310,8 +377,10 @@ def embed(**kwargs):
         cls = type(saved_shell_instance)
         cls.clear_instance()
     frame = sys._getframe(1)
-    shell = InteractiveShellEmbed.instance(_call_location_id='%s:%s' % (frame.f_code.co_filename, frame.f_lineno), **kwargs)
-    shell(header=header, stack_depth=2, compile_flags=compile_flags)
+    shell = InteractiveShellEmbed.instance(_init_location_id='%s:%s' % (
+        frame.f_code.co_filename, frame.f_lineno), **kwargs)
+    shell(header=header, stack_depth=2, compile_flags=compile_flags,
+          _call_location_id='%s:%s' % (frame.f_code.co_filename, frame.f_lineno))
     InteractiveShellEmbed.clear_instance()
     #restore previous instance
     if saved_shell_instance is not None:


Test patch: