Repository: ipython/ipython

Issue:
%run does not work with import hooks
Hi,

I ran into an issue where the use of `imp` vs `__import__` matters. `imp` is bypassing some things that import statements do, such as calling custom [import hooks](https://docs.python.org/3/reference/import.html#import-hooks). Is there are reason `imp` is being used? If not, can this be updated to use `__import__`?

As a note, this comes up when using `%run -m foo`.

Thanks!


Discussion:
Pull request?
This is a really vague bug report. We almost never use `imp` (it is deprecated) , though we do use  `importlib` and `__import__` in many places.  It will be hard to respond without narrowing what you wish / what your actual bug is. 

Sorry that I was not clear. Here is a reproducer:
IPython: `7.3.0`
Python: `3.7.3`

Code:
```
FULLPATH = '/tmp/my_tmp.py'
with open(FULLPATH, 'w') as f:
    f.write('''
print('Loaded my_tmp')
if __name__ == "__main__":
    print('I just ran a script')
''')
class MyTempImporter(object):
    def __init__(self):
        pass
 
    def find_module(self, fullname, path=None):
        if 'my_tmp' in fullname:
            return self
        return None
 
    def load_module(self, name):
        import imp
        return imp.load_source('my_tmp', FULLPATH)
    
    def get_code(self, fullname):
        import imp
        with open(FULLPATH, 'r') as f:
            compile(f.read(), 'foo', 'exec')
    def is_package(self, __):
        return False
 
import sys
sys.meta_path.insert(0, MyTempImporter())
```

```
In [2]:  %run -m my_tmp                                                                                                                                                                        
/usr/local/python/python-3.7/latest/lib64/python3.7/site-packages/IPython/core/magics/execution.py:683: UserWarning: 'my_tmp' is not a valid modulename on sys.path
  warn('%r is not a valid modulename on sys.path'%modulename)

In [3]: import my_tmp                                                                                                                                                                          
/usr/local/python/python-3.7/latest/bin/ipython:18: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
Loaded my_tmp
```

```
$ PYTHONPATH=/tmp python -m my_tmp
Loaded my_tmp
I just ran a script
```

I have opened this to fix the issue in `In[2]`. Please let me know if more info will help.
I'm confused; in the above code *you* are using `imp`, not us; so I'm not sure what you are asking about `imp`:

```
$ rg 'from imp '
```
```
IPython/extensions/autoreload.py
120:from imp import reload

IPython/utils/tests/test_path.py
15:from imp import reload
```
```
 ~/dev/ipython[master ✗] $ rg 'import imp$'
IPython/lib/deepreload.py
37:import imp
```

And those are extensions you do not seem to use, so not sure why `imp` would be involved in our codebase. 

As for `run -m`, it "simply" calls [runpy.run_module](https://docs.python.org/3/library/runpy.html#runpy.run_module), we may need to have some special handling for import hook. We can't use `__import__` as we don't import it, we run in a given namespace, and properly set some things like whether it is `__main__` or not.

I would have to re-read about importlib to understand how to fix things. 
> I'm confused; in the above code you are using imp, not us; so I'm not sure what you are asking about imp:

I copied some code, I didn't mean to imply that using imp is the right way to build this hook. Only to show how to make the error. Sorry for the additional confusion.


> As for run -m, it "simply" calls runpy.run_module, we may need to have some special handling for import hook. 

Thanks! It looks like I wrongly assumed the bug's cause. Given something similar to the above reproducer, using `%run` is what I am after having fixed. Please let me know if you need any more information. I have updated the title.
@mlucool when I tried to reproduce this just now using your example, I get:

```
$ PYTHONPATH=/tmp ipython
Python 3.7.3 (default, May 24 2019, 16:15:07)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: %run -m my_tmp
Loaded my_tmp
I just ran a script

In [2]: import my_tmp
Loaded my_tmp
```

This seems correct to me - is there something else that should be happening?
Ah, I see, I guess this is the real reproducer:

```
$ ipython
Python 3.7.3 (default, May 24 2019, 16:15:07)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: %run test.py # test.py is the code in https://github.com/ipython/ipython/issues/11736#issuecomment-494168069

In [2]: %run -m my_tmp
/home/goldbaum/Documents/ipython/IPython/core/magics/execution.py:675: UserWarning: 'my_tmp' is not a valid modulename on sys.path
  warn('%r is not a valid modulename on sys.path'%modulename)

In [3]: import my_tmp
Loaded my_tmp
```

Resolving patch:
diff --git a/IPython/core/magics/execution.py b/IPython/core/magics/execution.py
--- a/IPython/core/magics/execution.py
+++ b/IPython/core/magics/execution.py
@@ -692,6 +692,11 @@ def run(self, parameter_s='', runner=None,
                 warn('For Windows, use double quotes to wrap a filename: %run "mypath\\myfile.py"')
             error(msg)
             return
+        except TypeError:
+            if fpath in sys.meta_path:
+                filename = ""
+            else:
+                raise
 
         if filename.lower().endswith(('.ipy', '.ipynb')):
             with preserve_keys(self.shell.user_ns, '__file__'):
diff --git a/IPython/utils/module_paths.py b/IPython/utils/module_paths.py
--- a/IPython/utils/module_paths.py
+++ b/IPython/utils/module_paths.py
@@ -20,6 +20,7 @@
 # Stdlib imports
 import importlib
 import os
+import sys
 
 # Third-party imports
 
@@ -61,6 +62,8 @@ def find_mod(module_name):
     loader = importlib.util.find_spec(module_name)
     module_path = loader.origin
     if module_path is None:
+        if loader.loader in sys.meta_path:
+            return loader.loader
         return None
     else:
         split_path = module_path.split(".")


Test patch:
diff --git a/IPython/core/tests/test_magic.py b/IPython/core/tests/test_magic.py
--- a/IPython/core/tests/test_magic.py
+++ b/IPython/core/tests/test_magic.py
@@ -1198,3 +1198,49 @@ def test_timeit_arguments():
         # 3.7 optimize no-op statement like above out, and complain there is
         # nothing in the for loop.
         _ip.magic("timeit -n1 -r1 a=('#')")
+
+
+TEST_MODULE = """
+print('Loaded my_tmp')
+if __name__ == "__main__":
+    print('I just ran a script')
+"""
+
+
+def test_run_module_from_import_hook():
+    "Test that a module can be loaded via an import hook"
+    with TemporaryDirectory() as tmpdir:
+        fullpath = os.path.join(tmpdir, 'my_tmp.py')
+        with open(fullpath, 'w') as f:
+            f.write(TEST_MODULE)
+
+        class MyTempImporter(object):
+            def __init__(self):
+                pass
+
+            def find_module(self, fullname, path=None):
+                if 'my_tmp' in fullname:
+                    return self
+                return None
+
+            def load_module(self, name):
+                import imp
+                return imp.load_source('my_tmp', fullpath)
+
+            def get_code(self, fullname):
+                with open(fullpath, 'r') as f:
+                    return compile(f.read(), 'foo', 'exec')
+
+            def is_package(self, __):
+                return False
+
+        sys.meta_path.insert(0, MyTempImporter())
+
+        with capture_output() as captured:
+            _ip.magic("run -m my_tmp")
+            _ip.run_cell("import my_tmp")
+
+        output = "Loaded my_tmp\nI just ran a script\nLoaded my_tmp\n"
+        nt.assert_equal(output, captured.stdout)
+
+        sys.meta_path.pop(0)