Repository: microsoft/ptvsd

Issue:
Unable to add a breakpoint to a file containing an & in path name
## Environment data

- PTVSD version: Release
- OS and version: Windows, Powershell
- Python version (& distribution if applicable, e.g. Anaconda): N/A
- Using VS Code or Visual Studio:VSC

## Actual behavior

Debugger throws an error with `Unable to open .... file not found` error.

## Expected  behavior

Debugger should support `&` in path.

## Steps to reproduce:
1. Use Powershell as the shell in VSC
2. Create a folder with an `&`
3. Create a python file in that folder
4. Start debugging with a breakpoint in that file

Originally reported https://github.com/Microsoft/vscode-python/issues/2620
Paths are being incorrectly escaped
## Environment data

- PTVSD version: 4.1.4
- OS and version: Windows 10
- Python version (& distribution if applicable, e.g. Anaconda): CPython 3.7.0
- Using VS Code or Visual Studio: VSC

## Actual behavior

```
<module> (c:\Users\brcan\Desktop\Testing\minimal-django\mysite\settings.py:120)
_call_with_frames_removed (c:\users\brcan\desktop\testing\minimal-django\&lt;frozen importlib._bootstrap&gt;:219)
exec_module (c:\users\brcan\desktop\testing\minimal-django\&lt;frozen importlib._bootstrap_external&gt;:728)
_load_unlocked (c:\users\brcan\desktop\testing\minimal-django\&lt;frozen importlib._bootstrap&gt;:677)
_find_and_load_unlocked (c:\users\brcan\desktop\testing\minimal-django\&lt;frozen importlib._bootstrap&gt;:967)
_find_and_load (c:\users\brcan\desktop\testing\minimal-django\&lt;frozen importlib._bootstrap&gt;:983)
_gcd_import (c:\users\brcan\desktop\testing\minimal-django\&lt;frozen importlib._bootstrap&gt;:1006)
<module> (c:\Users\brcan\Desktop\Testing\minimal-django\manage.py:15)
```

## Expected  behavior

Do not escape `<` and `>` to `&lt;` and `&gt;`, respectively.

## Steps to reproduce:
See Microsoft/vscode-python#3063



Discussion:
@DonJayamanne do you have the error stack by chance?
Looking at the stack info that we receive from pydevd:
```xml
<xml>
    <thread id="pid_5276_id_2217585763272">
        <frame id="2217603071240" name="&lt;module&gt;" file="C%3A%5CGIT%5Cpyscratch2%5Ctest%26amp%3Bmytest%5Cmytest.py" line="1"></frame>
        <frame id="2217595381496" name="_run" file="C%3A%5CGIT%5Cptvsd%5Cptvsd%5C_local.py" line="140"></frame>
        <frame id="2217602507328" name="run_file" file="C%3A%5CGIT%5Cptvsd%5Cptvsd%5C_local.py" line="79"></frame>
        <frame id="2217601072432" name="debug_main" file="C%3A%5CGIT%5Cptvsd%5Cptvsd%5C_local.py" line="45"></frame>
        <frame id="2217601071960" name="handle_args" file="C%3A%5CGIT%5Cptvsd%5Cptvsd%5C__main__.py" line="220"></frame>
        <frame id="2217549471256" name="main" file="C%3A%5CGIT%5Cptvsd%5Cptvsd%5C__main__.py" line="227"></frame>
        <frame id="2217551981048" name="&lt;module&gt;" file="C%3A%5CUsers%5Ckanadig%5C.vscode%5Cextensions%5Cms-python.python-2018.10.0-alpha%5CpythonFiles%5Cexperimental%5Cptvsd_launcher.py" line="38"></frame>
    </thread>
</xml>
```
The folder name gets escaped xml escaped and url escaped. `test%26amp%3Bmytest`. Since we unquote after untangle has processed the xml we don't get the correct path back.
[Option 1] is to xml unescape after we unquote in `wrapper.py`
[Option 2] One option is to add `&;` to quote in `pydevd_comm.py`: https://github.com/Microsoft/ptvsd/blob/fcdb060b76d2648ae8788e4ac8e5097c4fbf24ec/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py#L820


Same issue as #948

Resolving patch:
diff --git a/ptvsd/wrapper.py b/ptvsd/wrapper.py
--- a/ptvsd/wrapper.py
+++ b/ptvsd/wrapper.py
@@ -17,6 +17,7 @@
 import sys
 import threading
 import traceback
+
 try:
     import urllib
     urllib.unquote
@@ -32,6 +33,7 @@
     import Queue as queue
 import warnings
 from xml.sax import SAXParseException
+from xml.sax.saxutils import unescape as xml_unescape
 
 import _pydevd_bundle.pydevd_comm as pydevd_comm  # noqa
 import _pydevd_bundle.pydevd_comm_constants as pydevd_comm_constants  # noqa
@@ -212,6 +214,15 @@ def unquote(s):
     return urllib.unquote(s)
 
 
+def unquote_xml_path(s):
+    """XML unescape after url unquote. This reverses the escapes and quotes done
+    by pydevd.
+    """
+    if s is None:
+        return None
+    return xml_unescape(unquote(str(s)))
+
+
 class IDMap(object):
     """Maps VSCode entities to corresponding pydevd entities by ID.
 
@@ -1686,7 +1697,7 @@ def on_stackTrace(self, request, args):
             fid = self.frame_map.to_vscode(key, autogen=True)
             name = unquote(xframe['name'])
             # pydevd encodes if necessary and then uses urllib.quote.
-            norm_path = self.path_casing.un_normcase(unquote(str(xframe['file'])))  # noqa
+            norm_path = self.path_casing.un_normcase(unquote_xml_path(xframe['file']))  # noqa
             source_reference = self.get_source_reference(norm_path)
             if not self.internals_filter.is_internal_path(norm_path):
                 module = self.modules_mgr.add_or_get_from_path(norm_path)
@@ -2253,7 +2264,7 @@ def _parse_exception_details(self, exc_xml, include_stack=True):
                 xframes = list(xml.thread.frame)
                 frame_data = []
                 for f in xframes:
-                    file_path = unquote(f['file'])
+                    file_path = unquote_xml_path(f['file'])
                     if not self.internals_filter.is_internal_path(file_path) \
                        and self._should_debug(file_path):
                         line_no = int(f['line'])
@@ -2270,7 +2281,7 @@ def _parse_exception_details(self, exc_xml, include_stack=True):
                                                func_name, None))
 
                 exc_stack = ''.join(traceback.format_list(frame_data))
-                exc_source = unquote(xframes[0]['file'])
+                exc_source = unquote_xml_path(xframes[0]['file'])
                 if self.internals_filter.is_internal_path(exc_source) or \
                     not self._should_debug(exc_source):
                     exc_source = None
@@ -2471,7 +2482,7 @@ def on_pydevd_thread_suspend(self, seq, args):
         # do step over and step out
         xframes = list(xml.thread.frame)
         xframe = xframes[0]
-        filepath = unquote(xframe['file'])
+        filepath = unquote_xml_path(xframe['file'])
         if reason in STEP_REASONS or reason in EXCEPTION_REASONS:
             if self.internals_filter.is_internal_path(filepath) or \
                 not self._should_debug(filepath):


Test patch:
diff --git a/pytests/func/test_breakpoints.py b/pytests/func/test_breakpoints.py
new file mode 100644
--- /dev/null
+++ b/pytests/func/test_breakpoints.py
@@ -0,0 +1,26 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See LICENSE in the project root
+# for license information.
+
+from __future__ import print_function, with_statement, absolute_import
+
+import os.path
+from pytests.helpers.timeline import Event
+from ..helpers.pathutils import get_test_root, compare_path
+
+BP_TEST_ROOT = get_test_root('bp')
+
+
+def test_path_with_ampersand(debug_session):
+    bp_line = 2
+    testfile = os.path.join(BP_TEST_ROOT, 'a&b', 'test.py')
+    debug_session.common_setup(testfile, 'file', [bp_line])
+    debug_session.start_debugging()
+    hit = debug_session.wait_for_thread_stopped()
+    frames = hit.stacktrace.body['stackFrames']
+    assert compare_path(frames[0]['source']['path'], testfile)
+
+    debug_session.send_request('continue').wait_for_response()
+    debug_session.wait_for_next(Event('continued'))
+
+    debug_session.wait_for_exit()
diff --git a/pytests/func/testfiles/bp/a&b/test.py b/pytests/func/testfiles/bp/a&b/test.py
new file mode 100644
--- /dev/null
+++ b/pytests/func/testfiles/bp/a&b/test.py
@@ -0,0 +1,3 @@
+print('one')
+print('two')
+print('three')
\ No newline at end of file