Repository: xonsh/xonsh

Issue:
Inconsistency importing modules between using ~/.xonshrc and ~/.config/xonsh/rc.xsh
I found that you can use `~/.config/xonsh/rc.xsh` instead of `~/.xonshrc` to configure xonsh, but when I tried to import a module on a separate file I found that while on `~/.xonshrc` this works as [detailed in the tutorial](https://xon.sh/tutorial.html#importing-xonsh-xsh), trying to do the same in `~/.config/xonsh/rc.xsh` fails. I was able to work around this by using the following snippet

```
with open(p"$HOME/.config/xonsh/my_module.xsh") as file:
    execx(file.read())
```

but I think this should not be the correct approach. I tried appending `~/.config/xonsh` to both `$PATH` and `$PYTHONPATH` on top of the run control file but that did not help.

## xonfig

<details>

```
$ xonfig
+------------------+----------------------+
| xonsh            | 0.9.24.dev1424       |
| Git SHA          | b4931a59             |
| Commit Date      | Oct 27 17:05:19 2020 |
| Python           | 3.8.5                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 3.0.5                |
| shell type       | prompt_toolkit       |
| pygments         | 2.7.1                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | pop                  |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
| on jupyter       | False                |
| jupyter kernel   | None                 |
| xontrib          | []                   |
+------------------+----------------------+
```

**Note:** The xonsh version is a build [this](https://github.com/dyuri/xonsh/tree/ptk_support_for_themes) branch because of a fix not yet merged.

</details>

## Expected Behaviour

Configuring the run control for xonsh should behave the same when using either `~/.xonshrc` or `~/.config/xonsh/rc.xsh`.

## Current Behaviour

When I run this on my home folder
```
$ echo 'print(1)' > my_module.xsh
$ echo 'from my_module import *' > .xonshrc
$ xonsh
```
then I get a `1` on stdout and the module is correctly imported, but if I run the following inside `~/.config/xonsh`
```
$ echo 'print(1)' > my_module.xsh
$ echo 'from my_module import *' > rc.xsh
$ xonsh
```
then I get the traceback detailed beneath.

**Note:** If I place the module on my home folder, then the import is successful, which led me to try to `cd` into `~/.config/xonsh/` before running xonsh but the behaviour was the same, so this does not depend on where xonsh is started, it seems to always look in the user's home folder.

### Traceback

<details>

```
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.8/site-packages/xonsh/__amalgam__.py", line 17286, in xonsh_script_run_control
    run_script_with_cache(filename, execer, ctx)
  File "/home/user/.local/lib/python3.8/site-packages/xonsh/__amalgam__.py", line 2471, in run_script_with_cache
    run_compiled_code(ccode, glb, loc, mode)
  File "/home/user/.local/lib/python3.8/site-packages/xonsh/__amalgam__.py", line 2376, in run_compiled_code
    func(code, glb, loc)
  File "/home/user/.config/xonsh/rc.xsh", line 9, in <module>
    from my_module import *
ModuleNotFoundError: No module named 'my_module'
error running xonsh run control file '/home/user/.config/xonsh/rc.xsh': No module named 'my_module'
```

</details>

## Steps to Reproduce

Running these commands seem to be the easiest way to reproduce:
```
$ echo 'print(1)' > ~/.config/xonsh/my_module.xsh
$ echo 'from my_module import *' > ~/.config/xonsh/rc.xsh
$ xonsh
```

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**



Discussion:
Thanks for reporting this @kip93! What is your $PYTHONPATH set to? This is really what determines what is importable.
Hello @scopatz! The `$PYTHONPATH` variable is not set, and even if I do `$PYTHONPATH = [p"$HOME/.config/xonsh/"]` before importing the module is still not found. I will update the issue to add this for clarity.
Ok, with some more testing I realised that while putting `$PYTHONPATH = [p"$HOME/.config/xonsh/"]` on top of `~/.config/xonsh/rc.xsh` does not work, running `$PYTHONPATH=[p"$HOME/.config/xonsh/"] xonsh` does seem to work. Still, I do not think that this should be the correct behaviour, but it at least provides a starting point.

Maybe, when importing from `~/.config/xonsh/rc.xsh` this value could be appended to the `$PYTHONPATH` by default? Or maybe we need to edit the working directory? That first approach seems simple enough, I could take a stab at it and do a PR. Editing the working dir on the other hand, I am not sure how we would do that without some potential nasty side effects.
Hi @kip93! This works for me:
```bash
# bash
echo 'from my_module import *' > $HOME/.config/xonsh/rc.xsh
echo 'print(1)' > $HOME/.config/xonsh/my_module.xsh
PYTHONPATH=$HOME/.config/xonsh xonsh 
# 1
# xonsh
```
or without PYTHONPATH setting:
```bash
#bash
rc="$HOME/.config/xonsh/rc.xsh"
echo "import sys" > $rc
echo "sys.path.append('$HOME/.config/xonsh')" >> $rc
echo "from my_module import *" >> $rc
echo 'print(1)' > ~/.config/xonsh/my_module.xsh
xonsh
# 1
# xonsh
```
Yeah, I can see an argument for adding these default rc directories to the PYTHONPATH, even if just temporarily while loading rc files. I think that would be a good fix. 
@anki-code That's what I clarified on my last comment, if you patch the `$PYTHONPATH` either by setting it before running xonsh or by using `sys.path.append` as per your example, this works. But, I would say that that is a workaround rather than correct behaviour.
@scopatz If patching the `$PYTHONPATH` while loading rc files is a good approach, then this weekend I can try to do it myself and create a PR.

I just have 2 questions to help me with this fix faster, since this is my first time hacking at xonsh:
1. Which file has the code responsible for start up and loading rc files, and
2. Which are all the possible directories that an rc file could be located at? I know of `~/.xonshrc`, `~/.config/xonsh/rc.xsh` and `/etc/xonshrc`, but these are the Linux file locations, not sure if they are exactly the same on BSD & OSx or if there is some difference there, and I have no clue where they are on Windows.

I could probably find that info if I look at the code but I do not want to potentially end up losing the weekend just to locate that info.
I believe that there is a `load_rcs()` function or similar somewhere.  The list of default RC files lives in `enviorn.py`. See https://xon.sh/envvars.html#xonshrc too, `$XONSHRC`
Thanks @scopatz! Those pointers are a really big help (:

I will work on this and let you guys know if and when I have a fix ready.
Hello again! As you can see above, I just opened a PR with a fix. Only really tested it on my linux machine, but nothing I used should cause any issues with Windows. Let me know what you think!
Well, it seems I was wrong, god damn Windows test do not pass ):

I will continue debugging and report back.
Hello there! It's me again. I wanted to put down some findings that I discovered while failing to fix Windows support.

Seems like on Windows, prior to version 3.9 of Python, the registry is used to handle the `sys.path` (see [this issue](https://bugs.python.org/issue8901)). I tried several workarounds, but none of them were able to correctly handle the case when an user might explicitly add the same path as the rc to `sys.path`, which I would consider to be a regression, since before this patch the user could do that without any problem.

This is made even worse because I cannot properly test any of this because the most modern Windows PC that I have access to is a 2005ish 32-bit Windows XP netbook which I do not know if it even boots anymore (and I do not have a Windows license to use on a VM), so I have been depending on the pytest checks' output.

Other than that, this has had no issue on the linux and macos tests. So much so, that with only the first 4 commits the PR was done were it not for Windows.

Unfortunately, I do not have any more time to pout into this this weekend. I will tackle this again next week.
I'm back!

During the week I remembered that you can get a copy of Windows 10 without a key for reduced functionality, so I spun up a VM and did some debugging. With this, I found that:

1. On Windows, any sort of wrapper I try to put on the `sys.path` is completely ignored (prior to Python 3.9), probably because of how Windows handles it using the registry.
2. Seems like the error with missing `sys.path` entries that are manually added in the rc file is not due to my PR. I copied the test to the master branch and the same behaviour is present, so it is not a regression as I originally thought. This probably deserves its own issue, since it only presented itself here because I added a test for this case.
To avoid the failing test I added a condition to skip if running on Windows with a version of Python prior to 3.9.

I believe that the PR is ready to be reviewed. Should I add a label or something to the PR?
I'll take a look at the PR again
I use the following setup and it helps me to avoid PATH problems https://gist.github.com/jnoortheen/8b5220f200bc5c9e121a7722836bf674

Resolving patch:
diff --git a/xonsh/environ.py b/xonsh/environ.py
--- a/xonsh/environ.py
+++ b/xonsh/environ.py
@@ -2219,7 +2219,6 @@ def xonshrc_context(rcfiles=None, execer=None, ctx=None, env=None, login=True):
         if not os.path.isfile(rcfile):
             loaded.append(False)
             continue
-        _, ext = os.path.splitext(rcfile)
         status = xonsh_script_run_control(rcfile, ctx, env, execer=execer, login=login)
         loaded.append(status)
     if env["THREAD_SUBPROCS"] is None:
@@ -2250,11 +2249,21 @@ def foreign_env_fixes(ctx):
         del ctx["PROMPT"]
 
 
+class _RcPath(str):
+    """A class used exclusively to know which entry was added temporarily to
+    sys.path while loading rc files.
+    """
+
+    pass
+
+
 def xonsh_script_run_control(filename, ctx, env, execer=None, login=True):
     """Loads a xonsh file and applies it as a run control."""
     if execer is None:
         return False
     updates = {"__file__": filename, "__name__": os.path.abspath(filename)}
+    rc_dir = _RcPath(os.path.dirname(filename))
+    sys.path.append(rc_dir)
     try:
         with swap_values(ctx, updates):
             run_script_with_cache(filename, execer, ctx)
@@ -2267,6 +2276,8 @@ def xonsh_script_run_control(filename, ctx, env, execer=None, login=True):
         msg = "error running xonsh run control file {0!r}: {1!s}"
         print_exception(msg.format(filename, err))
         loaded = False
+    finally:
+        sys.path = list(filter(lambda p: p is not rc_dir, sys.path))
     return loaded
 
 


Test patch:
diff --git a/tests/test_main.py b/tests/test_main.py
--- a/tests/test_main.py
+++ b/tests/test_main.py
@@ -12,7 +12,7 @@
 import xonsh.main
 from xonsh.main import XonshMode
 import pytest
-from tools import TEST_DIR, skip_if_on_windows
+from tools import ON_WINDOWS, TEST_DIR, VER_FULL, skip_if_on_windows
 
 
 def Shell(*args, **kwargs):
@@ -70,6 +70,88 @@ def test_premain_custom_rc(shell, tmpdir, monkeypatch):
     assert f.strpath in builtins.__xonsh__.env.get("XONSHRC")
 
 
+def test_rc_with_modules(shell, tmpdir, monkeypatch, capsys):
+    """Test that an RC file can load modules inside the same folder it is located in."""
+
+    monkeypatch.setattr(sys.stdin, "isatty", lambda: True)
+    monkeypatch.setitem(os.environ, "XONSH_CACHE_SCRIPTS", "False")
+
+    tmpdir.join("my_python_module.py").write("print('Hello,')")
+    tmpdir.join("my_xonsh_module.xsh").write("print('World!')")
+    rc = tmpdir.join("rc.xsh")
+    rc.write("from my_python_module import *\nfrom my_xonsh_module import *")
+    xonsh.main.premain(["--rc", rc.strpath])
+
+    assert rc.strpath in builtins.__xonsh__.env.get("XONSHRC")
+    assert (
+        builtins.__xonsh__.env.get("LOADED_RC_FILES")[
+            builtins.__xonsh__.env.get("XONSHRC").index(rc.strpath)
+        ]
+        is True
+    )
+
+    stdout, stderr = capsys.readouterr()
+    assert "Hello,\nWorld!" in stdout
+    assert len(stderr) == 0
+
+    # Check that the temporary rc's folder is not left behind on the path
+    assert tmpdir.strpath not in sys.path
+
+
+@pytest.mark.skipif(ON_WINDOWS and VER_FULL < (3, 9), reason="See https://github.com/xonsh/xonsh/issues/3936")
+def test_rc_with_modified_path(shell, tmpdir, monkeypatch, capsys):
+    """Test that an RC file can edit the sys.path variable without losing those values."""
+
+    monkeypatch.setattr(sys.stdin, "isatty", lambda: True)
+    monkeypatch.setitem(os.environ, "XONSH_CACHE_SCRIPTS", "False")
+
+    rc = tmpdir.join("rc.xsh")
+    rc.write(f"import sys\nsys.path.append('{tmpdir.strpath}')\nprint('Hello, World!')")
+    xonsh.main.premain(["--rc", rc.strpath])
+
+    assert rc.strpath in builtins.__xonsh__.env.get("XONSHRC")
+    assert (
+        builtins.__xonsh__.env.get("LOADED_RC_FILES")[
+            builtins.__xonsh__.env.get("XONSHRC").index(rc.strpath)
+        ]
+        is True
+    )
+
+    stdout, stderr = capsys.readouterr()
+    assert "Hello, World!" in stdout
+    assert len(stderr) == 0
+
+    # Check that the path that was explicitly added is not accidentally deleted
+    assert tmpdir.strpath in sys.path
+
+
+def test_rc_with_failing_module(shell, tmpdir, monkeypatch, capsys):
+    """Test that an RC file which imports a module that throws an exception ."""
+
+    monkeypatch.setattr(sys.stdin, "isatty", lambda: True)
+    monkeypatch.setitem(os.environ, "XONSH_CACHE_SCRIPTS", "False")
+
+    tmpdir.join("my_failing_module.py").write("raise RuntimeError('Unexpected error')")
+    rc = tmpdir.join("rc.xsh")
+    rc.write("from my_failing_module import *")
+    xonsh.main.premain(["--rc", rc.strpath])
+
+    assert rc.strpath in builtins.__xonsh__.env.get("XONSHRC")
+    assert (
+        builtins.__xonsh__.env.get("LOADED_RC_FILES")[
+            builtins.__xonsh__.env.get("XONSHRC").index(rc.strpath)
+        ]
+        is False
+    )
+
+    stdout, stderr = capsys.readouterr()
+    assert len(stdout) == 0
+    assert "RuntimeError: Unexpected error" in stderr
+
+    # Check that the temporary rc's folder is not left behind on the path
+    assert tmpdir.strpath not in sys.path
+
+
 def test_no_rc_with_script(shell, tmpdir):
     args = xonsh.main.premain(["tests/sample.xsh"])
     assert not (args.mode == XonshMode.interactive)