Repository: spyder-ide/spyder

Issue:
The ouline explorer doesn't populate for some files with latest 4.x
## Problem Description
The outline explorer just doesn't want to populate at all for some files on my system. Moreover, the code style and error analysis do not seem to work for these files too.

Here are some lsp logs that I have created by trying to open a very simple file for which the outline explorer doesn't work.

[transport_python_1716.log](https://github.com/spyder-ide/spyder/files/5330042/transport_python_1716.log)
[server_python_1716.log](https://github.com/spyder-ide/spyder/files/5330043/server_python_1716.log)

The content of the file was simply:

```python
# -*- coding: utf-8 -*-


def test_outline_explorer():
    a = 3
    c = b
    return a
```

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 4.2.0.dev0 6e0666389
* Python version: 3.7.6 64-bit
* Qt version: 5.12.5
* PyQt5 version: 5.12.3
* Operating System: Windows 10

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```

# Mandatory:
atomicwrites >=1.2.0           :  1.4.0 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  1.5.0 (OK)
diff_match_patch >=20181111    :  20200713 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.16.1 (OK)
jedi =0.17.2                   :  0.17.2 (OK)
nbconvert >=4.0                :  5.6.1 (OK)
numpydoc >=0.6.0               :  1.1.0 (OK)
paramiko >=2.4.0               :  2.7.1 (OK)
parso =0.7.0                   :  0.7.0 (OK)
pexpect >=4.4.0                :  4.8.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.7.2 (OK)
pygments >=2.0                 :  2.6.1 (OK)
pylint >=1.0                   :  2.5.3 (OK)
pyls >=0.35.0;<1.0.0           :  0.35.1+1.gd81c7ba (OK)
qdarkstyle >=2.8               :  2.8.1 (OK)
qtawesome >=0.5.7              :  0.8.0.dev0 (OK)
qtconsole >=4.7.7              :  4.8.0.dev0 (OK)
qtpy >=1.5.0                   :  1.9.0 (OK)
rtree >=0.8.3                  :  0.9.4 (OK)
sphinx >=0.6.6                 :  3.1.2 (OK)
spyder_kernels >=1.9.4;<1.10.0 :  1.10.0.dev0 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  19.0.1 (OK)

# Optional:
cython >=0.21                  :  0.29.21 (OK)
matplotlib >=2.0.0             :  3.2.2 (OK)
numpy >=1.7                    :  1.19.1 (OK)
pandas >=1.1.1                 :  1.0.5 (NOK)
scipy >=0.17.0                 :  1.5.0 (OK)
sympy >=0.7.3                  :  None (OK)
```



Discussion:
Opening the same file at commit 66e98a510d16a727e04a9e2a705102a9feb8016b, right before PR #13109 was merged, is working flawlessly.
@jnsebgosselin can you please check if the error persists if you update the PyLS repo after this PR https://github.com/palantir/python-language-server/pull/866 ?
Thanks for the followup @steff456.

The issue is still happening with latest python-language-server on the `develop` branch.

I don't know if this is related, but there is an error that is reported in the log files.

```python
2020-10-05 18:35:19,363 UTC - ERROR - pyls_jsonrpc.endpoint - Failed to handle notification textDocument/didOpen: {'textDocument': {'uri': 'file:///D:/test_outline_explorer.py', 'languageId': 'Python', 'version': 1, 'text': '# -*- coding: utf-8 -*-\n\n\ndef test_outline_explorer():\n    a = 3\n    c = b\n    return a\n'}}
Traceback (most recent call last):
  File "C:\Anaconda3\envs\python37\lib\site-packages\pyls_jsonrpc\endpoint.py", line 142, in _handle_notification
    handler_result = handler(params)
  File "C:\Anaconda3\envs\python37\lib\site-packages\pyls_jsonrpc\dispatchers.py", line 23, in handler
    return method(**(params or {}))
  File "C:\Users\User\python-language-server\pyls\python_ls.py", line 298, in m_text_document__did_open
    workspace.put_document(textDocument['uri'], textDocument['text'], version=textDocument.get('version'))
AttributeError: 'NoneType' object has no attribute 'put_document'
```

This error doesn't occur if I go before PR #13109 was merged on the 4.x branch.
What version of json-rpc do you have in the environment?
It is version 0.4.0.

![image](https://user-images.githubusercontent.com/10170372/95140013-d0a6f680-073b-11eb-8959-4b087198c8c4.png)

Thanks a lot @jnsebgosselin for your help with tracking this bug!

> I don't know if this is related, but there is an error that is reported in the log files.

I think this is related to an error while initializing the PyLS because (according to your traceback) the document is trying to be assigned to a non-existing workspace.

Could you try with PR #13828? That's ready for review and it should solve those issues.
Thanks @ccordoba12 for your help with this. I tried with PR #13828 and this doesn't solve the problem with the outline explorer not working for some files, but this solve the broken code analysis for those files and there isn't any traceback in the log files anymore...

What is very strange is that this seems to be related to the location of the files. I have folder that if I put any file in it, then Spyder is unable to build the outline explorer tree. If I take these files out and put them in another location, then everything seems to be working fine... this is really strange.
If I put the file `test_outline_explorer.py` in `C:\Users\User\gwhat\gwhat\meteo` it is working, but if I put it in `C:\Users\User\gwhat\gwhat\meteo\tests` it is not...

I will continue to investigate and report back if I find anything useful. 

![image](https://user-images.githubusercontent.com/10170372/95199351-b22c1400-07aa-11eb-8587-52f6232f6fb5.png)

ok, I don't know exactly what I did, but after restarting Spyder multiple times and deleting, renaming, copying, moving some files around, changing the root directory, the working directy, closing and opening my Spyder project, etc.... the problem seems to be fixed when running Spyder with PR #13828.

So maybe PR #13828 did fixed the problem after all and the problem was still occurring because of temp files or something? Anyway, I don't know. I suggest not to close this yet, I'll go through my day and I'll report back if the problem is indeed completely resolved or not.
>  tried with PR #13828 and ... this solve the broken code analysis for those files and there isn't any traceback in the log files anymore...

This is great news! Thanks for the confirmation @jnsebgosselin!

> Anyway, I don't know. I suggest not to close this yet, I'll go through my day and I'll report back if the problem is indeed completely resolved or not.

I was investigating this problem yesterday and I think I found the cause. I'm preparing a PR now and I'll ping you when it's up.
@ccordoba12 after working in Spyder all day using PR #13828, I cannot reproduce this bug anymore. So I guess  PR #13828 did indeed fix this issue.

The reason why I was still experiencing this issue after switching to PR #13828 and restarting Spyder might be because of some caching mechanism that eventually got resolved after I restarted Spyder several times and deleted/renamed/moved/restored the problematic folders?

In all cases, I cannot reproduce this problem anymore, so I think this issue should be closed now, or after PR #13828 is merged. It is as you wish.
This is going to be fixed by PR #13938 because even after the fixes in #13828, it's true that the Outline doesn't populate for files in your previous session (except the one that has focus when Spyder starts).
Here I was talking about files for which the outline explorer did not work at all, even after triggering a document change manually and even if the file was focused on startup.

Maybe I was not clear in my opening post, but it was not related to the issue where the outline tree was not populated automatically for the files that were opened in the previous session. This particular issue, I reported in https://github.com/spyder-ide/spyder/issues/13897.
Just to clarify my last post. I was experiencing two clearly distinct bugs with the outline explorer.

The first one was that the outline explorer was not populating the outline tree **automatically** for the files that were opened in the previous session. That I reported in Issue #13897, which I think is going to be fixed by PR #13938.

The second bug, which I reported here, was that no matter what I did, for some files, I was not able to get the outline explorer to work. This specific issue I cannot reproduce anymore after working all day with PR #13828.
OK, I can still reproduce the issue even when using PR #13828, but I have new information that could help to pinpoint the issue.

The issue seems to happen only when a Spyder project is active. When it is the case, then the outline explorer just cannot populate the tree for some files. The wheel is spinning endlessly.

If I close the project, then the outline explorer is able to populate the tree for these same files without any problem.

![outline_explorer_not_working_in_project](https://user-images.githubusercontent.com/10170372/95279608-21dce600-0821-11eb-968d-daf087379e1f.gif)

> The first one was that the outline explorer was not populating the outline tree automatically for the files that were opened in the previous session. That I reported in Issue #13897, which I think is going to be fixed by PR #13938.

Sorry, I thought this issue was about that problem. And yes, it's going to be fixed by PR #13938.

> The issue seems to happen only when a Spyder project is active. When it is the case, then the outline explorer just cannot populate the tree for some files. The wheel is spinning endlessly.

This should be solved by the combined effect of PRs #13828 and #13938.

So do you want to keep this one open until those PRs are merged?
Yes I would like to wait before closing this. I'm not yet convinced this is going to be fixed by PRs #13828 and #13938.
Here is a reproducible case with two Spyder files. You need to have a spyder project for the repo `Spyder` active for the bug to happen.

![image](https://user-images.githubusercontent.com/10170372/95345553-d78f4f80-0888-11eb-9bd2-cbc31682b432.png)

And that's even with both my PRs applied?
> 
> 
> And that's even with both my PRs applied?

Yes. The only difference is that the wheel stop spinning after a second or two, but the tree is never populated.
Ok, I'll take a look at that once my PRs are merged.
@ccordoba12 for the Spyder reproducible case, if I had a `__init__.py` file in the folder `spyder/tools`, then the outline explorer is able to build the outline tree for the files `create_app.py` and `rm_whitespace.py`. A restart is required after you add or remove the `__init__.py` file.
I am able to produce a very simple example for reproducing the issue.

Basically, all you need to do is:

1. Create a new Spyder project
2. Add a file with some functions in the newly created project folder.
3. Create a new sub-folder.
4. Add a new file in the subfolder.
5. Open both files in Spyder.

The outline tree for the file that is in the subfolder is not going to be populated unless you add a `__init__.py` in the subfolder and restart spyder.

![image](https://user-images.githubusercontent.com/10170372/95358860-833f9c00-0897-11eb-8564-256f1cb734b9.png)



> The outline tree for the file that is in the subfolder is not going yo be populated unless you add a __init__.py in the subfolder and restart spyder.

This looks like an issue with Jedi or the PyLS, so not an easy fix. But thanks for the reproducible example.
> This looks like an issue with Jedi or the PyLS, so not an easy fix. But thanks for the reproducible example.

Ok , this is not good news :/

Let me know if I can help further with this issue. I've tried to look more into it, but unfortunately, all that pyls stuff is a little bit over my head...

Resolving patch:
diff --git a/external-deps/python-language-server/pyls/plugins/symbols.py b/external-deps/python-language-server/pyls/plugins/symbols.py
--- a/external-deps/python-language-server/pyls/plugins/symbols.py
+++ b/external-deps/python-language-server/pyls/plugins/symbols.py
@@ -1,5 +1,7 @@
 # Copyright 2017 Palantir Technologies, Inc.
 import logging
+import os
+
 from pyls import hookimpl
 from pyls.lsp import SymbolKind
 
@@ -15,7 +17,13 @@ def pyls_document_symbols(config, document):
     symbols_settings = config.plugin_settings('jedi_symbols')
     all_scopes = symbols_settings.get('all_scopes', True)
     add_import_symbols = symbols_settings.get('include_import_symbols', True)
-    definitions = document.jedi_names(all_scopes=all_scopes)
+
+    use_document_path = False
+    document_dir = os.path.normpath(os.path.dirname(document.path))
+    if not os.path.isfile(os.path.join(document_dir, '__init__.py')):
+        use_document_path = True
+
+    definitions = document.jedi_names(use_document_path, all_scopes=all_scopes)
     module_name = document.dot_path
     symbols = []
     exclude = set({})
diff --git a/external-deps/python-language-server/pyls/workspace.py b/external-deps/python-language-server/pyls/workspace.py
--- a/external-deps/python-language-server/pyls/workspace.py
+++ b/external-deps/python-language-server/pyls/workspace.py
@@ -234,8 +234,8 @@ def word_at_position(self, position):
         return m_start[0] + m_end[-1]
 
     @lock
-    def jedi_names(self, all_scopes=False, definitions=True, references=False):
-        script = self.jedi_script()
+    def jedi_names(self, use_document_path, all_scopes=False, definitions=True, references=False):
+        script = self.jedi_script(use_document_path=use_document_path)
         return script.get_names(all_scopes=all_scopes, definitions=definitions,
                                 references=references)
 
@@ -263,7 +263,7 @@ def jedi_script(self, position=None, use_document_path=False):
 
         # Extend sys_path with document's path if requested
         if use_document_path:
-            sys_path += [os.path.dirname(self.path)]
+            sys_path += [os.path.normpath(os.path.dirname(self.path))]
 
         kwargs = {
             'code': self.source,
diff --git a/tools/rm_whitespace.py b/tools/rm_whitespace.py
--- a/tools/rm_whitespace.py
+++ b/tools/rm_whitespace.py
@@ -1,11 +1,9 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# -----------------------------------------------------------------------------
 # Copyright (c) 2009- Spyder Project Contributors
 #
 # Distributed under the terms of the MIT License
 # (see spyder/__init__.py for details)
-# -----------------------------------------------------------------------------
 
 """Script and functions to automatically remove trailing spaces from files."""
 


Test patch:
diff --git a/spyder/app/tests/test_mainwindow.py b/spyder/app/tests/test_mainwindow.py
--- a/spyder/app/tests/test_mainwindow.py
+++ b/spyder/app/tests/test_mainwindow.py
@@ -16,13 +16,10 @@
 import os.path as osp
 import re
 import shutil
+import sys
 import tempfile
 from textwrap import dedent
-try:
-    from unittest.mock import Mock, MagicMock
-except ImportError:
-    from mock import Mock, MagicMock  # Python 2
-import sys
+from unittest.mock import Mock, MagicMock
 import uuid
 
 # Third party imports
@@ -36,11 +33,11 @@
 import pylint
 import pytest
 from qtpy import PYQT5, PYQT_VERSION
-from qtpy.QtCore import Qt, QTimer, QEvent, QPoint, QUrl
+from qtpy.QtCore import Qt, QTimer, QUrl
 from qtpy.QtTest import QTest
 from qtpy.QtGui import QImage
 from qtpy.QtWidgets import (QAction, QApplication, QFileDialog, QLineEdit,
-                            QTabBar, QToolTip, QWidget)
+                            QTabBar, QWidget)
 from qtpy.QtWebEngineWidgets import WEBENGINE
 
 # Local imports
@@ -49,7 +46,6 @@
 from spyder.app.mainwindow import MainWindow  # Tests fail without this import
 from spyder.config.base import get_home_dir, get_conf_path, get_module_path
 from spyder.config.manager import CONF
-from spyder.widgets.dock import TabFilter
 from spyder.preferences.runconfig import RunConfiguration
 from spyder.plugins.base import PluginWindow
 from spyder.plugins.help.widgets import ObjectComboBox
@@ -58,16 +54,8 @@
 from spyder.plugins.projects.projecttypes import EmptyProject
 from spyder.py3compat import PY2, to_text_string
 from spyder.utils.misc import remove_backslashes
-from spyder.utils.programs import is_module_installed
 from spyder.widgets.dock import DockTitleBar
 
-# For testing various Spyder urls
-if not PY2:
-    from urllib.request import urlopen
-    from urllib.error import URLError
-else:
-    from urllib2 import urlopen, URLError
-
 
 # =============================================================================
 # ---- Constants
@@ -1610,8 +1598,6 @@ def test_c_and_n_pdb_commands(main_window, qtbot):
 @pytest.mark.skipif(os.name == 'nt', reason="It times out sometimes on Windows")
 def test_stop_dbg(main_window, qtbot):
     """Test that we correctly stop a debugging session."""
-    nsb = main_window.variableexplorer.get_focus_widget()
-
     # Wait until the window is fully up
     shell = main_window.ipyconsole.get_current_shellwidget()
     qtbot.waitUntil(lambda: shell._prompt_html is not None, timeout=SHELL_TIMEOUT)
@@ -2580,7 +2566,6 @@ def test_go_to_definition(main_window, qtbot, capsys):
 
     # Create new editor with code and wait until LSP is ready
     main_window.editor.new(text=code_def)
-    n_editors = len(main_window.editor.get_filenames())
     code_editor = main_window.editor.get_focus_widget()
     with qtbot.waitSignal(code_editor.lsp_response_signal, timeout=30000):
         code_editor.document_did_open()
@@ -2657,7 +2642,6 @@ def test_runcell(main_window, qtbot, tmpdir, debug):
     # Execute runcell
     with qtbot.waitSignal(shell.executed):
         shell.execute(function + u"(0, r'{}')".format(to_text_string(p)))
-    control = main_window.ipyconsole.get_focus_widget()
 
     if debug:
         # Reach the 'name' input
@@ -3371,7 +3355,6 @@ def test_varexp_cleared_after_kernel_restart(main_window, qtbot):
     Test that the variable explorer is cleared after a kernel restart.
     """
     shell = main_window.ipyconsole.get_current_shellwidget()
-    control = main_window.ipyconsole.get_focus_widget()
     qtbot.waitUntil(lambda: shell._prompt_html is not None,
                     timeout=SHELL_TIMEOUT)
 
@@ -3401,7 +3384,6 @@ def test_varexp_cleared_after_reset(main_window, qtbot):
     reset in the IPython console and variable explorer panes.
     """
     shell = main_window.ipyconsole.get_current_shellwidget()
-    control = main_window.ipyconsole.get_focus_widget()
     qtbot.waitUntil(lambda: shell._prompt_html is not None,
                     timeout=SHELL_TIMEOUT)
 
@@ -3745,5 +3727,29 @@ def test_continue_first_line(main_window, qtbot):
     assert "b = 9" in shell._control.toPlainText()
 
 
+@pytest.mark.slow
+@flaky(max_runs=3)
+@pytest.mark.use_introspection
+@pytest.mark.skipif(os.name == 'nt', reason="Fails on Windows")
+def test_outline_no_init(main_window, qtbot):
+    # Open file in one of our directories without an __init__ file
+    spy_dir = osp.dirname(get_module_path('spyder'))
+    main_window.editor.load(osp.join(spy_dir, 'tools', 'rm_whitespace.py'))
+
+    # Show outline explorer
+    outline_explorer = main_window.outlineexplorer
+    outline_explorer._toggle_view_action.setChecked(True)
+
+    # Wait a bit for trees to be filled
+    qtbot.wait(5000)
+
+    # Get tree length
+    treewidget = outline_explorer.explorer.treewidget
+    editor_id = list(treewidget.editor_ids.values())[1]
+
+    # Assert symbols in the file are detected and shown
+    assert len(treewidget.editor_tree_cache[editor_id]) > 0
+
+
 if __name__ == "__main__":
     pytest.main()