Repository: apple/swift-docc (swift)

Issue:
`convert` and `preview` commands crash on certain Markdown pages
### Description

When rendering Swift compiler documentation the diagnostics formatting code crashes on certain Markdown pages.

### Checklist

- [X] If possible, I've reproduced the issue using the `main` branch of this package.
- [X] This issue hasn't been addressed in an [existing GitHub issue](https://github.com/apple/swift-docc/issues).

### Expected Behavior

No crashes.

### Actual behavior

`Swift/StringCharacterView.swift:158: Fatal error: String index is out of bounds`

### Steps To Reproduce

1. Clone https://github.com/apple/swift.git
2. Install a recent development snapshot of Swift, I'm using `swift-DEVELOPMENT-SNAPSHOT-2023-07-23-a`.
3. Run DocC from this snapshot this way (assuming `swift` is the clone from step 1):
```
docc preview --allow-arbitrary-catalog-directories swift/docs
```

### Swift-DocC Version Information

swift-DEVELOPMENT-SNAPSHOT-2023-07-23-a

### Swift Compiler Version Information

```shell
Apple Swift version 5.9-dev (LLVM 9b562f55c38e378, Swift b4ee68bd37c4f7d)
Target: arm64-apple-macosx14.0
```

Pull request:
Bug/issue #, if applicable:  https://github.com/apple/swift-docc/issues/678

## Summary

Previously, the default diagnostic formatter would crash due to incorrect indices handling when trying to highlight a diagnostic source range. 

A diagnostic's `SourceRange` is the range in the source file that emits the diagnostic. When highlighting a diagnostic source range, we use the `column` property of the `upperBound` and `lowerBound`  to find indices of a source string that should be highlighted. 

The `column` property is The number of bytes in UTF-8 encoding.
>     /// The number of bytes in UTF-8 encoding from the start of the line to the character at this source location.
However, this was not taken into account previously when calculating the indices. 

This pull request fixes this crash by using proper UTF-8 indices. 

## Testing

This functionally can be tested using the steps mentioned in https://github.com/apple/swift-docc/issues/678.

1. Clone https://github.com/apple/swift.git
2. Install a recent development snapshot of Swift
3. Run DocC from this snapshot this way (assuming swift is the clone from step 1): 
`docc preview --allow-arbitrary-catalog-directories swift/docs`

Alternatively, the issue is easily reproducible by adding an emoji to a source line that would emit a diagnostic. 

## Checklist

Make sure you check off the following items. If they cannot be completed, provide a reason.

- [x] Added tests
- [x] Ran the `./bin/test` script and it succeeded
- [x] Updated documentation if necessary

Patch:
diff --git a/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticConsoleWriter.swift b/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticConsoleWriter.swift
index e454323f86..133f753c0f 100644
--- a/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticConsoleWriter.swift
+++ b/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticConsoleWriter.swift
@@ -443,17 +443,20 @@ extension DefaultDiagnosticConsoleFormatter {
         } else {
             endColumn = sourceLine.count + 1
         }
-        
+
+        let sourceLineUTF8 = sourceLine.utf8
+
         let columnRange = startColumn..<endColumn
-        let startIndex = sourceLine.index(sourceLine.startIndex, offsetBy: columnRange.lowerBound - 1)
-        let endIndex = sourceLine.index(startIndex, offsetBy: columnRange.count)
+        let startIndex = sourceLineUTF8.index(sourceLineUTF8.startIndex, offsetBy: columnRange.lowerBound - 1)
+        let endIndex = sourceLineUTF8.index(startIndex, offsetBy: columnRange.count)
+        let highlightRange = startIndex..<endIndex
         
         let ansiAnnotation = ANSIAnnotation.sourceHighlight
 
         var result = ""
-        result += sourceLine[sourceLine.startIndex..<startIndex]
-        result += ansiAnnotation.applied(to: String(sourceLine[startIndex..<endIndex]))
-        result += sourceLine[endIndex..<sourceLine.endIndex]
+        result += sourceLine[sourceLine.startIndex..<highlightRange.lowerBound]
+        result += ansiAnnotation.applied(to: String(sourceLine[highlightRange]))
+        result += sourceLine[highlightRange.upperBound..<sourceLine.endIndex]
         
         return result
     }


Test patch:
diff --git a/Tests/SwiftDocCTests/Diagnostics/DiagnosticConsoleWriterDefaultFormattingTest.swift b/Tests/SwiftDocCTests/Diagnostics/DiagnosticConsoleWriterDefaultFormattingTest.swift
index 3a68a11550..ca359d9941 100644
--- a/Tests/SwiftDocCTests/Diagnostics/DiagnosticConsoleWriterDefaultFormattingTest.swift
+++ b/Tests/SwiftDocCTests/Diagnostics/DiagnosticConsoleWriterDefaultFormattingTest.swift
@@ -245,6 +245,35 @@ class DiagnosticConsoleWriterDefaultFormattingTest: XCTestCase {
         """)
     }
 
+    func testDisplaysSource_WithEmojis_ProperlyHighlightsSource() {
+        let identifier = "org.swift.docc.test-identifier"
+        let summary = "Test diagnostic summary"
+        let explanation = "Test diagnostic explanation."
+        let baseURL =  Bundle.module.url(
+            forResource: "TestBundle", withExtension: "docc", subdirectory: "Test Bundles")!
+        let source = baseURL.appendingPathComponent("TestTutorial.tutorial")
+        let range = SourceLocation(line: 39, column: 4, source: source)..<SourceLocation(line: 39, column: 53, source: source)
+
+        let logger = Logger()
+        let consumer = DiagnosticConsoleWriter(logger, baseURL: baseURL, highlight: true)
+
+        let diagnostic = Diagnostic(source: source, severity: .warning, range: range, identifier: identifier, summary: summary, explanation: explanation)
+        let problem = Problem(diagnostic: diagnostic, possibleSolutions: [])
+        consumer.receive([problem])
+        try? consumer.finalize()
+        print(logger.output)
+        XCTAssertEqual(logger.output, """
+        \u{001B}[1;33mwarning: \(summary)\u{001B}[0;0m
+        \(explanation)
+          --> TestTutorial.tutorial:39:4-39:53
+        37 |    }
+        38 |    
+        39 +    \u{001B}[1;32m@Section(title: "Create a New AR Project 💻") {\u{001B}[0;0m
+        40 |       @ContentAndMedia {
+        41 |          Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt
+        """)
+    }
+
     func testDisplaysPossibleSolutionsSummary() {
         let identifier = "org.swift.docc.test-identifier"
         let summary = "Test diagnostic summary"