Repository: platers/obsidian-linter (ts)

Issue:
FR: tidy footnotes
 for footnote, deliberately does *not* preserve the relation between key and footnote
maybe this plugin is a good reference:https://github.com/charliecm/obsidian-tidy-footnotes
Feature Request: Sherlock tidy-footnotes plugin?
I tried adding the command from the [tidy footnotes plugin](https://obsidian.md/plugins?id=obsidian-tidy-footnotes), as an extra custom command. However, it seems the plugin leaves extra blank lines at the end of a file if there are no footnotes in a note. even with the rule enforcing exactly one line break at the end of a file, somehow there are still the extra blank lines added at the end.

Theoretically, this could be an issue for the tidy footnotes plugin, but it does not look very active anymore. Moreover, the linter already has footnote linting rules, making the addition actually useful for users not looking for extra plugins. So the tidy footnote code could maybe be simply sherlocked for a simpler (and probably more bug free) user setup?
FR: re-index-footnotes Should not Overwrite Duplicate References
## Describe the Bug

`re-index-footnotes` doesn't seem to allow multiple references to the same footnote. Using footnotes for references will often lead to situations where the same footnote is used again and again, so the text has multiple of the same footnote reference. The Re-Index Footnotes rule simply changes the duplicate numbers into false, non-existent footnote references.

## How to Reproduce

Steps to reproduce the behavior:

1. Create a file with the following text:
```markdown
First,[^1] followed by the Second,[^2] and then the First[^1] again.

[^1]: First Reference
[^2]: Second reference
```
2. Run Linter with rule `re-index-footnotes` enabled.
3. The text above is changed to the following:
```markdown
First,[^1] followed by the Second,[^2] and then the First[^3] again.

[^1]: First Reference
[^2]: Second reference
```

## Expected Behavior

Leave the second `[^1]` reference alone, as it is a valid reference to the first footnote.

## Screenshots

If applicable, add screenshots to help explain your problem.

## Device

- [x] Desktop
- [ ] Mobile



Pull request description:
Swap Re-Index from a Merged File Use Case to a Just Re-Index Use Case
Fixes #625 
Fixes #359 
Fixes #466 

The rule for re-indexing footnotes was originally added for the purpose of merging two files and then de-duping the footnotes that had the same key in a 1 to 1 manner. However there is a desire to move the logic over to a 1 to many option where there is 1 footnote for many keys. This allows for preserving the footnote to key relationship while also making things a little clearer as to how they work.

Changes Made:
- Added a couple of tests for the scenarios provided on the relevant FRs and Bug report
- Updated the wording of the re-index rule to make it clear that the functionality has changed
- Updated the logic of re-index to re-order the footnotes and de-dupe footnotes based on the content in the files
- Made sure to throw an error if there was more than 1 distinct footnote for a key since that makes it really hard to know which footnote goes with which key(s)
- Removed a test that no longer makes sense and updated another to make it clear that multiple footnote keys are now allowed
- Fixed a variable typo
- Updated the logic to have another strings helper for replacing a value starting at a particular index in a string since that was needed for proper indexing in this case

Patch:
diff --git a/src/lang/locale/en.ts b/src/lang/locale/en.ts
index c2999b7..2085f73 100644
--- a/src/lang/locale/en.ts
+++ b/src/lang/locale/en.ts
@@ -77,6 +77,7 @@ export default {
 
     // mdast.ts
     'missing-footnote-error-message': `Footnote '{FOOTNOTE}' has no corresponding footnote reference before the footnote contents and cannot be processed. Please make sure that all footnotes have a corresponding reference before the content of the footnote.`,
+    'too-many-footnotes-error-message': `Footnote key '{FOOTNOTE_KEY}' has more than 1 footnote referencing it. Please update the footnotes so that there is only one footnote per footnote key.`,
 
     // rules.ts
     'wrapper-yaml-error': 'error in the yaml: {ERROR_MESSAGE}',
@@ -465,7 +466,7 @@ export default {
     // re-index-footnotes.ts
     're-index-footnotes': {
       'name': 'Re-Index Footnotes',
-      'description': 'Re-indexes footnote keys and footnote, based on the order of occurrence (NOTE: This rule deliberately does *not* preserve the relation between key and footnote, to be able to re-index duplicate keys.)',
+      'description': 'Re-indexes footnote keys and footnote, based on the order of occurrence (NOTE: This rule does *not* work if there is more than one footnote for a key.)',
     },
     // remove-consecutive-list-markers.ts
     'remove-consecutive-list-markers': {
diff --git a/src/rules/re-index-footnotes.ts b/src/rules/re-index-footnotes.ts
index 59a0351..a9070af 100644
--- a/src/rules/re-index-footnotes.ts
+++ b/src/rules/re-index-footnotes.ts
@@ -2,6 +2,7 @@ import {Options, RuleType} from '../rules';
 import RuleBuilder, {ExampleBuilder, OptionBuilderBase} from './rule-builder';
 import dedent from 'ts-dedent';
 import {ignoreListOfTypes, IgnoreTypes} from '../utils/ignore-types';
+import {reIndexFootnotes} from '../utils/mdast';
 
 class ReIndexFootnotesOptions implements Options {}
 
@@ -18,23 +19,7 @@ export default class ReIndexFootnotes extends RuleBuilder<ReIndexFootnotesOption
     return ReIndexFootnotesOptions;
   }
   apply(text: string, options: ReIndexFootnotesOptions): string {
-    return ignoreListOfTypes([IgnoreTypes.code, IgnoreTypes.inlineCode, IgnoreTypes.math, IgnoreTypes.yaml, IgnoreTypes.link, IgnoreTypes.wikiLink, IgnoreTypes.tag], text, (text) => {
-      // re-index footnote-text
-      let ft_index = 0;
-      text = text.replace(/^\[\^\w+\]: /gm, function() {
-        ft_index++;
-        return '[^' + String(ft_index) + ']: ';
-      });
-
-      // re-index footnote-keys
-      // regex uses hack to treat lookahead as lookaround https://stackoverflow.com/a/43232659
-      ft_index = 0;
-      text = text.replace(/(?!^)\[\^\w+\]/gm, function() {
-        ft_index++;
-        return '[^' + String(ft_index) + ']';
-      });
-      return text;
-    });
+    return ignoreListOfTypes([IgnoreTypes.code, IgnoreTypes.inlineCode, IgnoreTypes.math, IgnoreTypes.yaml, IgnoreTypes.link, IgnoreTypes.wikiLink, IgnoreTypes.tag], text, reIndexFootnotes);
   }
   get exampleBuilders(): ExampleBuilder<ReIndexFootnotesOptions>[] {
     return [
@@ -71,18 +56,35 @@ export default class ReIndexFootnotes extends RuleBuilder<ReIndexFootnotesOption
         `,
       }),
       new ExampleBuilder({
-        description: 'Re-indexing duplicate footnote keys',
+        description: 'Re-indexing footnotes preserves multiple references to the same footnote index',
         before: dedent`
-          Lorem ipsum at aliquet felis.[^1] Donec dictum turpis quis pellentesque,[^1] et iaculis tortor condimentum.
+          Lorem ipsum dolor sit amet, consectetur adipiscing elit.[^1] Aenean at aliquet felis. Donec dictum turpis quis ipsum pellentesque, et iaculis tortor condimentum.[^1a] Vestibulum nec blandit felis, vulputate finibus purus.[^2] Praesent quis iaculis diam.[^1]
           ${''}
           [^1]: first footnote
-          [^1]: second footnote
+          [^1a]: third footnote, inserted later
+          [^2]: second footnotes
         `,
         after: dedent`
-          Lorem ipsum at aliquet felis.[^1] Donec dictum turpis quis pellentesque,[^2] et iaculis tortor condimentum.
+          Lorem ipsum dolor sit amet, consectetur adipiscing elit.[^1] Aenean at aliquet felis. Donec dictum turpis quis ipsum pellentesque, et iaculis tortor condimentum.[^2] Vestibulum nec blandit felis, vulputate finibus purus.[^3] Praesent quis iaculis diam.[^1]
           ${''}
           [^1]: first footnote
-          [^2]: second footnote
+          [^2]: third footnote, inserted later
+          [^3]: second footnotes
+        `,
+      }),
+      new ExampleBuilder({ // accounts for https://github.com/platers/obsidian-linter/issues/466
+        description: 'Re-indexing footnotes condense duplicate footnotes into 1 when key and footnote are the same',
+        before: dedent`
+          bla[^1], bla[^1], bla[^2]
+          [^1]: bla
+          [^1]: bla
+          [^2]: bla
+        `,
+        after: dedent`
+          bla[^1], bla[^1], bla[^2]
+          ${''}
+          [^1]: bla
+          [^2]: bla
         `,
       }),
     ];
diff --git a/src/utils/mdast.ts b/src/utils/mdast.ts
index 9340f6b..d1af02d 100644
--- a/src/utils/mdast.ts
+++ b/src/utils/mdast.ts
@@ -1,7 +1,7 @@
 import {visit} from 'unist-util-visit';
 import type {Position} from 'unist';
 import type {Root} from 'mdast';
-import {hashString53Bit, makeSureContentHasEmptyLinesAddedBeforeAndAfter, replaceTextBetweenStartAndEndWithNewValue, getStartOfLineIndex} from './strings';
+import {hashString53Bit, makeSureContentHasEmptyLinesAddedBeforeAndAfter, replaceTextBetweenStartAndEndWithNewValue, getStartOfLineIndex, replaceAt} from './strings';
 import {genericLinkRegex} from './regex';
 import {gfmFootnote} from 'micromark-extension-gfm-footnote';
 import {gfmTaskListItem} from 'micromark-extension-gfm-task-list-item';
@@ -98,7 +98,7 @@ export function getPositions(type: MDAstTypes, text: string): Position[] {
  * @param {string} text The text to move footnotes in
  * @return {string} The text with footnote declarations moved to the end
  */
-export function moveFootnotesToEnd(text: string) {
+export function moveFootnotesToEnd(text: string): string {
   const positions: Position[] = getPositions(MDAstTypes.Footnote, text);
   let footnotes: string[] = [];
 
@@ -111,7 +111,7 @@ export function moveFootnotesToEnd(text: string) {
   const footnoteKeyToFootnoteKeyInfo = new Map<string, footnoteKeyInfo>();
   const mapOfFootnoteToFootnoteReferenceIndex = new Map<string, number>();
 
-  const getAllReferencePositionsForFootnote = function(footnote: string, startOfFootnoteReferenceSearch: number): void {
+  const getAllReferencePositionsForFootnote = function(text: string, footnote: string, startOfFootnoteReferenceSearch: number): void {
     const footnoteReference = footnote.match(/\[\^.*?\]/)[0];
 
     if (footnoteKeyToFootnoteKeyInfo.has(footnoteReference)) {
@@ -145,20 +145,7 @@ export function moveFootnotesToEnd(text: string) {
     footnoteKeyToFootnoteKeyInfo.set(footnoteReference, keyInfo);
   };
 
-  for (const position of positions) {
-    const footnote = text.substring(position.start.offset, position.end.offset);
-    footnotes.push(footnote);
-    // Remove the newline after the footnote if it exists
-    if (position.end.offset < text.length && text[position.end.offset] === '\n') {
-      text = text.substring(0, position.end.offset) + text.substring(position.end.offset + 1);
-    }
-    // Remove the newline after the footnote if it exists
-    if (position.end.offset < text.length && text[position.end.offset] === '\n') {
-      text = text.substring(0, position.end.offset) + text.substring(position.end.offset + 1);
-    }
-    text = text.substring(0, position.start.offset) + text.substring(position.end.offset);
-    getAllReferencePositionsForFootnote(footnote, position.start.offset);
-  }
+  text = removeFootnotesAndDoAnActionThem(positions, text, footnotes, getAllReferencePositionsForFootnote);
 
   for (const footnoteData of footnoteKeyToFootnoteKeyInfo) {
     const keyInfo = footnoteData[1];
@@ -191,6 +178,114 @@ export function moveFootnotesToEnd(text: string) {
   return text;
 }
 
+/**
+ * Re-indexes the footnotes in the document making sure that they increase in number from 1 on up
+ * and reorders the footnotes themselves as well.
+ * @param {string} text - The text to re-index the footnotes in.
+ * @return {string} The text with footnotes re-indexed.
+ */
+export function reIndexFootnotes(text: string): string {
+  const positions: Position[] = getPositions(MDAstTypes.Footnote, text);
+  let footnotes: string[] = [];
+
+  const mapOfFootnoteToFirstFootnoteReferenceIndex = new Map<string, number>();
+  const footnoteToFootnoteKey = new Map<string, string>();
+  const oldKeyToNewKey = new Map<string, string>();
+  let footnoteReferenceLocationInfo: {key: string, position: number}[] = [];
+  const footnoteKeys = new Set<string>();
+
+  const getFirstReferenceToFootnote = function(text: string, footnote: string, startOfFootnoteReferenceSearch: number): number {
+    const footnoteReference = footnote.match(/\[\^.*?\]/)[0];
+    footnoteToFootnoteKey.set(footnote, footnoteReference);
+
+    const footnoteKeyAlreadyUsed = footnoteKeys.has(footnoteReference);
+    if (footnoteKeyAlreadyUsed && mapOfFootnoteToFirstFootnoteReferenceIndex.has(footnote)) {
+      return mapOfFootnoteToFirstFootnoteReferenceIndex.get(footnote);
+    } else if (footnoteKeyAlreadyUsed) {
+      throw new Error(getTextInLanguage('logs.too-many-footnotes-error-message').replace('{FOOTNOTE_KEY}', footnoteReference));
+    }
+
+    let footnoteReferenceLocation: number;
+    let firstFootnoteReferenceIndex: number = -1;
+    do {
+      footnoteReferenceLocation = text.lastIndexOf(footnoteReference, startOfFootnoteReferenceSearch);
+      if (footnoteReferenceLocation === -1) {
+        continue;
+      }
+
+      footnoteReferenceLocationInfo.push({key: footnoteReference, position: footnoteReferenceLocation});
+      firstFootnoteReferenceIndex = footnoteReferenceLocation;
+      startOfFootnoteReferenceSearch = footnoteReferenceLocation - 1;
+    } while (footnoteReferenceLocation > -1);
+
+    footnoteKeys.add(footnoteReference);
+
+    return firstFootnoteReferenceIndex;
+  };
+
+  text = removeFootnotesAndDoAnActionThem(positions, text, footnotes, (text: string, footnote: string, startOfFootnoteReferenceSearch: number) => {
+    mapOfFootnoteToFirstFootnoteReferenceIndex.set(footnote, getFirstReferenceToFootnote(text, footnote, startOfFootnoteReferenceSearch));
+  });
+
+  // Sort the footnotes into the order of their references in the text
+  footnotes = footnotes.sort((f1: string, f2: string) => {
+    return mapOfFootnoteToFirstFootnoteReferenceIndex.get(f1) - mapOfFootnoteToFirstFootnoteReferenceIndex.get(f2);
+  });
+
+  // Sort the footnote references from last to first to prevent issues when replacing the footnote references down the road
+  footnoteReferenceLocationInfo = footnoteReferenceLocationInfo.sort((f1: {key: string, position: number}, f2: {key: string, position: number}) => {
+    return f2.position - f1.position;
+  });
+
+  // Add the footnotes to the end of the document
+  if (footnotes.length > 0) {
+    text = text.trimEnd() + '\n';
+  }
+
+  let footnoteIndex = 1;
+  const footnotesAdded = new Set<string>();
+  for (const footnote of footnotes) {
+    if (footnotesAdded.has(footnote)) {
+      continue;
+    }
+
+    footnotesAdded.add(footnote);
+    const footnoteKey = footnoteToFootnoteKey.get(footnote);
+    const newFootnoteKey = `[^${footnoteIndex++}]`;
+    oldKeyToNewKey.set(footnoteKey, newFootnoteKey);
+
+    text += '\n' + footnote.replace(footnoteKey, newFootnoteKey);
+  }
+
+  for (const footnoteReference of footnoteReferenceLocationInfo) {
+    const newFootnoteKey = oldKeyToNewKey.get(footnoteReference.key);
+
+    text = replaceAt(text, footnoteReference.key, newFootnoteKey, footnoteReference.position);
+  }
+
+  return text;
+}
+
+function removeFootnotesAndDoAnActionThem(positions: Position[], text: string, footnotes: string[], action: (text: string, footnote: string, startOfFootnoteReferenceSearch: number) => void) {
+  for (const position of positions) {
+    const footnote = text.substring(position.start.offset, position.end.offset);
+    footnotes.push(footnote);
+    // Remove the newline after the footnote if it exists
+    if (position.end.offset < text.length && text[position.end.offset] === '\n') {
+      text = text.substring(0, position.end.offset) + text.substring(position.end.offset + 1);
+    }
+    // Remove the newline after the footnote if it exists
+    if (position.end.offset < text.length && text[position.end.offset] === '\n') {
+      text = text.substring(0, position.end.offset) + text.substring(position.end.offset + 1);
+    }
+    text = text.substring(0, position.start.offset) + text.substring(position.end.offset);
+
+    action(text, footnote, position.start.offset);
+  }
+
+  return text;
+}
+
 /**
  * Makes sure that the style of either strong or emphasis is consistent.
  * @param {string} text The text to style either the strong or emphasis in a consistent manner
@@ -527,8 +622,8 @@ export function updateOrderedListItemIndicators(text: string, orderedListStyle:
       const listItemIndicatorLevel = getListItemLevel($1);
       // when dealing with a value that is not an int reset all values greater than or equal to the current list level
       if (!/^\d/.test($3)) {
-        const highestCurretnValue = listItemIndicatorLevel > lastItemListIndicatorLevel ? listItemIndicatorLevel: lastItemListIndicatorLevel;
-        removeListItemsItemIndicatorInfo(listItemIndicatorLevel, highestCurretnValue);
+        const highestCurrentValue = listItemIndicatorLevel > lastItemListIndicatorLevel ? listItemIndicatorLevel: lastItemListIndicatorLevel;
+        removeListItemsItemIndicatorInfo(listItemIndicatorLevel, highestCurrentValue);
 
         return listItem; // skip to the next item if the current item is not an ordered list item
       }
diff --git a/src/utils/strings.ts b/src/utils/strings.ts
index 0046d72..ea758fa 100644
--- a/src/utils/strings.ts
+++ b/src/utils/strings.ts
@@ -283,3 +283,20 @@ export function getStartOfLineIndex(text: string, indexToStartFrom: number): num
 
   return startOfLineIndex;
 }
+
+/**
+ * Replace the first instance of the matching search string in the text after the provided starting position.
+ * @param {string} text - The text in which to do the find and replace given the starting position.
+ * @param {string} search - The text to search for and replace in the provided string.
+ * @param {string} replace - The text to replace the search text with in the provided string.
+ * @param {number} start - The position to start the replace search at.
+ * @return {string} The new string after replacing the value if found.
+ */
+export function replaceAt(text: string, search: string, replace: string, start: number): string {
+  if (start > text.length - 1) {
+    return text;
+  }
+
+  return text.slice(0, start) +
+      text.slice(start, text.length).replace(search, replace);
+}


Test patch:
diff --git a/__tests__/re-index-footnotes.test.ts b/__tests__/re-index-footnotes.test.ts
index e149cde..6d458b5 100644
--- a/__tests__/re-index-footnotes.test.ts
+++ b/__tests__/re-index-footnotes.test.ts
@@ -14,6 +14,35 @@ ruleTest({
         \`h[^ae]llo\`
       `,
     },
+    { // accounts for https://github.com/platers/obsidian-linter/issues/359
+      testName: 'Out of order references are sorted into their proper our based on the order in which they first appear in the file',
+      before: dedent`
+        aaaaaa[^1]
+        cccccc[^3]
+        ddddd[^4]
+        bbbbb[^2]
+        fffffff[^5]
+        ${''}
+        [^1]: a-footnote
+        [^2]: b-footnote
+        [^3]: c-footnote
+        [^4]: d-footnote
+        [^5]: f-footnote
+      `,
+      after: dedent`
+        aaaaaa[^1]
+        cccccc[^2]
+        ddddd[^3]
+        bbbbb[^4]
+        fffffff[^5]
+        ${''}
+        [^1]: a-footnote
+        [^2]: c-footnote
+        [^3]: d-footnote
+        [^4]: b-footnote
+        [^5]: f-footnote
+      `,
+    },
   ],
 });