Repository: jackdewinter/pymarkdown

Issue:
BadTokenizationError: reference style link in header
## Bug Report

Please replace any of `[these areas]` in the paragraphs below with the requested information.

### Bug Type

*  Assertion Failure?
* Scan/Rule not working as expected

### Description

`BadTokenizationError` exception thrown during file processing

### Specifics

What operating system and version are you running into this behavior on?

* Oracle Linux Server release 8.9

What version are you seeing this behavior in? (Run `pip list` or `pipenv run pip list` and look for the entry beside `pymarkdownlnt`.)

* 0.9.19 
* (note: 0.9.18 and before work as expected)

Are there any extra steps that need to be taken before executing the application?

* none

What is the command line you invoke to get this behavior?

```
# normal
scan test-ref.md
# with extended logging
--log-level DEBUG scan test-ref.md
```

Are you using a configuration file? Either on the command line or one of the implicit configuration files? If so, attach that file to this issue.

* [.pymarkdown.yml](https://github.com/jackdewinter/pymarkdown/files/15329359/pymarkdown-config.txt)

What Markdown document causes this behavior to manifest?  Attach that file to this issue.

```
# test

## Example: [foo][bar]

[foo]: https://example.com
```

* [test-ref.md](https://github.com/jackdewinter/pymarkdown/files/15329378/test-ref.md)


### Actual Behavior

* `BadTokenizationError`
* [test-ref.log](https://github.com/jackdewinter/pymarkdown/files/15329387/test-ref.log)

### Expected Behavior

* Processed file

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/jackdewinter/pymarkdown/pull/1094?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 100.00%. Comparing base [(`e4ef5c6`)](https://app.codecov.io/gh/jackdewinter/pymarkdown/commit/e4ef5c6b193ad3f5013e8a7721a0379e07cc1178?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) to head [(`043cd5c`)](https://app.codecov.io/gh/jackdewinter/pymarkdown/commit/043cd5c04f25e8d86aedf920802aa9a1c115ea01?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).


<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##              main     #1094   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          189       189           
  Lines        19207     19206    -1     
  Branches      2394      2394           
=========================================
- Hits         19207     19206    -1     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/jackdewinter/pymarkdown/pull/1094?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).


Patch:
diff --git a/newdocs/src/changelog.md b/newdocs/src/changelog.md
index 118b9945..432393b6 100644
--- a/newdocs/src/changelog.md
+++ b/newdocs/src/changelog.md
@@ -13,7 +13,10 @@
 
 ### Fixed
 
-- None
+- [Issue 1015](https://github.com/jackdewinter/pymarkdown/issues/1015)
+    - Fixed issue with double tabs within fenced block
+- [Issue 1077](https://github.com/jackdewinter/pymarkdown/issues/1077)
+    - Fixed issue with previous cleanup
 
 ### Changed
 
diff --git a/publish/coverage.json b/publish/coverage.json
index 58abf6da..4f942223 100644
--- a/publish/coverage.json
+++ b/publish/coverage.json
@@ -6,8 +6,8 @@
         "totalCovered": 4789
     },
     "lineLevel": {
-        "totalMeasured": 19207,
-        "totalCovered": 19207
+        "totalMeasured": 19206,
+        "totalCovered": 19206
     }
 }
 
diff --git a/pymarkdown/links/link_parse_helper.py b/pymarkdown/links/link_parse_helper.py
index 825093e2..d1228331 100644
--- a/pymarkdown/links/link_parse_helper.py
+++ b/pymarkdown/links/link_parse_helper.py
@@ -148,7 +148,7 @@ class LinkParseHelper:
         new_index: int,
         source_text: str,
         text_from_blocks: str,
-        tabified_text: str,
+        tabified_text: Optional[str],
     ) -> Tuple[str, bool, int, str, str, Optional[str]]:
         POGGER.debug("collapsed reference?")
 
@@ -962,9 +962,6 @@ class LinkParseHelper:
         elif ParserHelper.is_character_at_index(
             source_text, new_index, LinkParseHelper.__link_format_reference_start
         ):
-            assert (
-                tabified_text is not None
-            ), "If we have a link start character, must have tabified text."
             (
                 lhp.label_type,
                 tried_full_reference_form,


Test patch:
diff --git a/publish/test-results.json b/publish/test-results.json
index 896c70b9..c459316a 100644
--- a/publish/test-results.json
+++ b/publish/test-results.json
@@ -1620,7 +1620,7 @@
         },
         {
             "name": "test.test_markdown_extra",
-            "totalTests": 138,
+            "totalTests": 143,
             "failedTests": 0,
             "errorTests": 0,
             "skippedTests": 1,
diff --git a/test/test_markdown_extra.py b/test/test_markdown_extra.py
index 6e9138d6..39cc3418 100644
--- a/test/test_markdown_extra.py
+++ b/test/test_markdown_extra.py
@@ -5747,6 +5747,182 @@ def test_extra_042d():
     act_and_assert(source_markdown, expected_gfm, expected_tokens)
 
 
+@pytest.mark.gfm
+def test_extra_042x():
+    """
+    TBD
+    """
+
+    # Arrange
+    source_markdown = """# test
+
+## Example: [foo][bar]
+
+[foo]: https://example.com
+"""
+    expected_tokens = [
+        "[atx(1,1):1:0:]",
+        "[text(1,3):test: ]",
+        "[end-atx::]",
+        "[BLANK(2,1):]",
+        "[atx(3,1):2:0:]",
+        "[text(3,4):Example: : ]",
+        "[text(3,13):[:]",
+        "[text(3,14):foo:]",
+        "[text(3,17):]:]",
+        "[text(3,18):[:]",
+        "[text(3,19):bar:]",
+        "[text(3,22):]:]",
+        "[end-atx::]",
+        "[BLANK(4,1):]",
+        "[link-ref-def(5,1):True::foo:: :https://example.com:::::]",
+        "[BLANK(6,1):]",
+    ]
+    expected_gfm = """<h1>test</h1>
+<h2>Example: [foo][bar]</h2>"""
+
+    # Act & Assert
+    act_and_assert(source_markdown, expected_gfm, expected_tokens)
+
+
+@pytest.mark.gfm
+def test_extra_042xx():
+    """
+    TBD
+    """
+
+    # Arrange
+    source_markdown = """# test
+
+## Example: [foo][bar]
+
+[bar]: https://example.com
+"""
+    expected_tokens = [
+        "[atx(1,1):1:0:]",
+        "[text(1,3):test: ]",
+        "[end-atx::]",
+        "[BLANK(2,1):]",
+        "[atx(3,1):2:0:]",
+        "[text(3,4):Example: : ]",
+        "[link(3,13):full:https://example.com::::bar:foo:False::::]",
+        "[text(3,14):foo:]",
+        "[end-link::]",
+        "[end-atx::]",
+        "[BLANK(4,1):]",
+        "[link-ref-def(5,1):True::bar:: :https://example.com:::::]",
+        "[BLANK(6,1):]",
+    ]
+    expected_gfm = """<h1>test</h1>
+<h2>Example: <a href="https://example.com">foo</a></h2>"""
+
+    # Act & Assert
+    act_and_assert(source_markdown, expected_gfm, expected_tokens)
+
+
+@pytest.mark.gfm
+def test_extra_042xa():
+    """
+    TBD
+    """
+
+    # Arrange
+    source_markdown = """# test
+
+## Example: [foo](https://example.com)
+
+"""
+    expected_tokens = [
+        "[atx(1,1):1:0:]",
+        "[text(1,3):test: ]",
+        "[end-atx::]",
+        "[BLANK(2,1):]",
+        "[atx(3,1):2:0:]",
+        "[text(3,4):Example: : ]",
+        "[link(3,13):inline:https://example.com:::::foo:False::::]",
+        "[text(3,14):foo:]",
+        "[end-link::]",
+        "[end-atx::]",
+        "[BLANK(4,1):]",
+        "[BLANK(5,1):]",
+    ]
+    expected_gfm = """<h1>test</h1>
+<h2>Example: <a href="https://example.com">foo</a></h2>"""
+
+    # Act & Assert
+    act_and_assert(source_markdown, expected_gfm, expected_tokens)
+
+
+@pytest.mark.gfm
+def test_extra_042xb():
+    """
+    TBD
+    """
+
+    # Arrange
+    source_markdown = """# test
+
+## Example: [foo]
+
+[foo]: https://example.com
+"""
+    expected_tokens = [
+        "[atx(1,1):1:0:]",
+        "[text(1,3):test: ]",
+        "[end-atx::]",
+        "[BLANK(2,1):]",
+        "[atx(3,1):2:0:]",
+        "[text(3,4):Example: : ]",
+        "[link(3,13):shortcut:https://example.com:::::foo:False::::]",
+        "[text(3,14):foo:]",
+        "[end-link::]",
+        "[end-atx::]",
+        "[BLANK(4,1):]",
+        "[link-ref-def(5,1):True::foo:: :https://example.com:::::]",
+        "[BLANK(6,1):]",
+    ]
+    expected_gfm = """<h1>test</h1>
+<h2>Example: <a href="https://example.com">foo</a></h2>"""
+
+    # Act & Assert
+    act_and_assert(source_markdown, expected_gfm, expected_tokens)
+
+
+@pytest.mark.gfm
+def test_extra_042xc():
+    """
+    TBD
+    """
+
+    # Arrange
+    source_markdown = """# test
+
+## Example: [foo][]
+
+[foo]: https://example.com
+"""
+    expected_tokens = [
+        "[atx(1,1):1:0:]",
+        "[text(1,3):test: ]",
+        "[end-atx::]",
+        "[BLANK(2,1):]",
+        "[atx(3,1):2:0:]",
+        "[text(3,4):Example: : ]",
+        "[link(3,13):collapsed:https://example.com:::::foo:False::::]",
+        "[text(3,14):foo:]",
+        "[end-link::]",
+        "[end-atx::]",
+        "[BLANK(4,1):]",
+        "[link-ref-def(5,1):True::foo:: :https://example.com:::::]",
+        "[BLANK(6,1):]",
+    ]
+    expected_gfm = """<h1>test</h1>
+<h2>Example: <a href="https://example.com">foo</a></h2>"""
+
+    # Act & Assert
+    act_and_assert(source_markdown, expected_gfm, expected_tokens)
+
+
 @pytest.mark.gfm
 def test_extra_999():
     """