Repository: psf/black

Issue:
Crash on docstring with tabs
**Describe the bug** A clear and concise description of what the bug is.

**To Reproduce** Steps to reproduce the behavior:

1. Take this file (in text)
2. Run _Black_ on it without any arguments
3. See error

**Expected behavior** Work

**Environment (please complete the following information):**
```bash
error: cannot format /home/user/cilantropy/dist_worker.py: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /tmp/blk_rifw8eif.log
Oh no! 💥 💔 💥
10 files left unchanged, 1 file failed to reformat.
```

- Version: 19.10b0
- OS and Python version: Linux (arch linux)/Python 3.8.3

**Does this bug also happen on master?** To answer this, you have two options:
pip install git+git://github.com/psf/black

**Additional context:**
Log:
```bash
--- src
+++ dst
@@ -62,11 +62,11 @@
       body=
         Expr(
           value=
             Constant(
               value=
-                'main worker for run subproccess and return result or error\n\n:param cmd: full command for run\t\n:return: result string and status error',  # str
+                'main worker for run subproccess and return result or error\n\n:param cmd: full command for run\n:return: result string and status error',  # str
             )  # /Constant
         )  # /Expr
         Try(
           body=
             Assign(
```
and my code:
```python
def main_worker(cmd):
	""" 
	main worker for run subproccess and return result or error

	:param cmd: full command for run	
	:return: result string and status error
	"""	

	try:
		cmd_response = subprocess.run(
			cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True
		)
	except subprocess.CalledProcessError as e:
		return "The pip command did not succeed: {stderr}".format(
				stderr=e.stderr.decode("utf-8")
			), True
		
	return cmd_response.stdout.decode("utf-8").replace(" "*6,"\n"), False
```




Discussion:
This is caused by the tab at the end of a line in the docstring:

![image](https://user-images.githubusercontent.com/1324225/90248816-12a17380-de42-11ea-9449-4688e1a80a37.png)

---

Here's a minimal example (initial tabs replaced with spaces):

```python
def thing():
    """ 
    line ending in a tab	
    line ending with no whitespace
    """
```

![image](https://user-images.githubusercontent.com/1324225/90248997-5f854a00-de42-11ea-8ee8-bd7af4025c9a.png)

---

Error does **not** occur with a tab at the end of the docstring:

```python
def thing():
    """ 
    line ending in a tab	
    """
```
And does **not** occur with a space at the end:

```python
def thing():
    """ 
    line ending in a space 
    line ending with no whitespace
    """
```

@hugovk thanks!
You're welcome! 

This is still a bug in Black, so we can keep it open to be investigated :)
`git bisect` points to a4c11a75e12300abfbe4c36854e450d42bdd1ee7 as the first bad commit, from "Re-indent the contents of docstrings" (#1053).
We can probably fix this by loosening the constraints on docstrings when we do the AST comparison, similar to what I did in #1593.
Updated (again), sorry for the noise.

Confirming a duplicate like #1638 (which may be a different but related issue in black 20.8b1)
Note that the bug also occurs when there's a literal tab in the middle of a line. It seems black converts that to spaces and errors out.

```python
def foo():
    """
    a	b  # it's a literal tab between a and b
    """
    pass
```

```console
$ black foo.py
error: cannot format foo.py: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /tmp/blk_q5uiqxs8.log
Oh no! 💥 💔 💥
1 file failed to reformat.
```

The contents of `/tmp/blk_q5uiqxs8.log`

```diff
--- src
+++ dst
@@ -23,11 +23,11 @@
       body=
         Expr(
           value=
             Constant(
               value=
-                "a\tb  # it's a literal tab between a and b",  # str
+                "a   b  # it's a literal tab between a and b",  # str
             )  # /Constant
         )  # /Expr
         Pass(
         )  # /Pass
       decorator_list=
```
Note also the bug only occurs with a multiline docstring:
This fails:
```python
def f(p):
    """p:	Parameter
    """
    pass
```
This passes:
```python
def f(p):
    """p:	Parameter"""
    pass
```
This appears to be caused by the intentional decision (introduced in #1053) to expand tabs to space (using `tabsize=8``, which is the [expandtabs()](https://docs.python.org/3.8/library/stdtypes.html#str.expandtabs) default):

https://github.com/psf/black/blob/6dddbd72414061cde9dd8ee72eac373b7fcf8b54/src/black/__init__.py#L6745-L6747

Resolving patch:
diff --git a/src/black/__init__.py b/src/black/__init__.py
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -6744,13 +6744,33 @@ def is_docstring(leaf: Leaf) -> bool:
     return False
 
 
+def lines_with_leading_tabs_expanded(s: str) -> List[str]:
+    """
+    Splits string into lines and expands only leading tabs (following the normal
+    Python rules)
+    """
+    lines = []
+    for line in s.splitlines():
+        # Find the index of the first non-whitespace character after a string of
+        # whitespace that includes at least one tab
+        match = re.match(r"\s*\t+\s*(\S)", line)
+        if match:
+            first_non_whitespace_idx = match.start(1)
+
+            lines.append(
+                line[:first_non_whitespace_idx].expandtabs()
+                + line[first_non_whitespace_idx:]
+            )
+        else:
+            lines.append(line)
+    return lines
+
+
 def fix_docstring(docstring: str, prefix: str) -> str:
     # https://www.python.org/dev/peps/pep-0257/#handling-docstring-indentation
     if not docstring:
         return ""
-    # Convert tabs to spaces (following the normal Python rules)
-    # and split into a list of lines:
-    lines = docstring.expandtabs().splitlines()
+    lines = lines_with_leading_tabs_expanded(docstring)
     # Determine minimum indentation (first line doesn't count):
     indent = sys.maxsize
     for line in lines[1:]:


Test patch:
diff --git a/tests/data/docstring.py b/tests/data/docstring.py
--- a/tests/data/docstring.py
+++ b/tests/data/docstring.py
@@ -110,6 +110,32 @@ def ignored_docstring():
     """a => \
 b"""  
 
+
+def docstring_with_inline_tabs_and_space_indentation():
+    """hey
+
+    tab	separated	value
+    	tab at start of line and then a tab	separated	value
+    				multiple tabs at the beginning	and	inline
+    	 	  	mixed tabs and spaces at beginning. next line has mixed tabs and spaces only.
+    			 	  		
+    line ends with some tabs		
+    """
+
+
+def docstring_with_inline_tabs_and_tab_indentation():
+	"""hey
+
+	tab	separated	value
+		tab at start of line and then a tab	separated	value
+					multiple tabs at the beginning	and	inline
+		 	  	mixed tabs and spaces at beginning. next line has mixed tabs and spaces only.
+				 	  		
+	line ends with some tabs		
+	"""
+	pass
+        
+
 # output
 
 class MyClass:
@@ -222,4 +248,29 @@ def believe_it_or_not_this_is_in_the_py_stdlib():
 
 def ignored_docstring():
     """a => \
-b"""
\ No newline at end of file
+b"""
+
+
+def docstring_with_inline_tabs_and_space_indentation():
+    """hey
+
+    tab	separated	value
+        tab at start of line and then a tab	separated	value
+                                multiple tabs at the beginning	and	inline
+                        mixed tabs and spaces at beginning. next line has mixed tabs and spaces only.
+
+    line ends with some tabs
+    """
+
+
+def docstring_with_inline_tabs_and_tab_indentation():
+    """hey
+
+    tab	separated	value
+            tab at start of line and then a tab	separated	value
+                                    multiple tabs at the beginning	and	inline
+                            mixed tabs and spaces at beginning. next line has mixed tabs and spaces only.
+
+    line ends with some tabs
+    """
+    pass