Repository: dandavison/delta (Rust)

Issue:
🐛 Include submodule details for all diff.submodule values
There are a few locations where submodule status is unexpectedly masked and git output is not included, but one would expect to be shown in some way. All of the tests below were using delta v0.8.3. In addition to the specific cases highlighted here, might it make sense to consider passing through lines not matching any delta pattern handling as a fallback for other similar cases?

When running `git-diff` (for working copy) or `git-log` (for commit history) using available values for the `diff.submodule` option, various output is unexpectedly generated.

## `diff.submodule=short`

This is the default git configuration and is handled reasonably enough, but a more terse output would be welcome. Something similar to the `diff.submodule=log` output header would be great, although that may subjectively vary too much from standard git output, and/or ambiguity in parsing. I'd probably direct my gripe most to why git's output is different for the short case rather than expect delta to handle it. Additionally, the "(rewind)" suffix isn't detected/added from git which is very useful, but similarly appears to be a git consistency issue and something possibly not in the realm of delta's responsibility.

Raw git output
```
diff --git a/submoduleA b/submoduleA
index 9caac0f..a2879d8 160000
--- a/submoduleA
+++ b/submoduleA
@@ -1 +1 @@
-Subproject commit 9caac0f494c9d0421aacf8d14a780b206c2bf19d
+Subproject commit a2879d8453f5b5bae5bf011078ba57ff75f6e44c
diff --git a/submoduleB b/submoduleB
index 70b6f0d..955ce29 160000
--- a/submoduleB
+++ b/submoduleB
@@ -1 +1 @@
-Subproject commit 70b6f0d1d2266803312bf6be2b2cc7aba670cdf6
+Subproject commit 955ce29f678cf180d8e73ae8ad1c0e5de3fd1d8a
diff --git a/submoduleC b/submoduleC
index 9662270..74dc49a 160000
--- a/submoduleC
+++ b/submoduleC
@@ -1 +1 @@
-Subproject commit 9662270c8cb24ff9386190acffb5cc9620858d78
+Subproject commit 74dc49aa4b02d278a83c983a2be398c44f157b84
```
Current `delta --no-gitconfig` default output
```
submoduleA
───────────────────────────────────────────

───┐
1: │
───┘
Subproject commit 9caac0f494c9d0421aacf8d14a780b206c2bf19d
Subproject commit a2879d8453f5b5bae5bf011078ba57ff75f6e44c

submoduleB
───────────────────────────────────────────

───┐
1: │
───┘
Subproject commit 70b6f0d1d2266803312bf6be2b2cc7aba670cdf6
Subproject commit 955ce29f678cf180d8e73ae8ad1c0e5de3fd1d8a

submoduleC
───────────────────────────────────────────

───┐
1: │
───┘
Subproject commit 9662270c8cb24ff9386190acffb5cc9620858d78
Subproject commit 74dc49aa4b02d278a83c983a2be398c44f157b84
```

Sample possible delta output, changes of a) removing "1:" block since line number isn't particularly useful, and b) condensing the previous and new commit into a single line as a commit range.
```
submoduleA
───────────────────────────────────────────

commit 9caac0f..a2879d8

submoduleB
───────────────────────────────────────────

commit 70b6f0d..955ce29

submoduleC
───────────────────────────────────────────

commit 9662270..74dc49a
```

## `diff.submodule=log`

This is my current individual configuration and the motivating case for writing up the issue. It has a significant impact because the commit difference list is simply excluded. I'm not sure that I'd expect delta to anything in such cases to abbreviate the output, but it should be passed through.

Raw git output:
```
Submodule submoduleA 9caac0f..a2879d8 (rewind):
  < Submodule A extra change 2
  < Submodule A extra change 1
  < Submodule A initial commit 4
  < Submodule A initial commit 3
Submodule submoduleB 70b6f0d..955ce29:
  > Submodule B stage change 3
  > Submodule B stage change 2
  > Submodule B stage change 1
Submodule submoduleC 9662270...74dc49a:
  < Submodule C extra change 2
  > Submodule C stage change 3
  < Submodule C extra change 1
  > Submodule C stage change 2
  > Submodule C stage change 1
```

Current `delta --no-gitconfig` default output:
```
Submodule submoduleA 9caac0f..a2879d8 (rewind):
─────────────────────────────────

Submodule submoduleB 70b6f0d..955ce29:
─────────────────────────────

Submodule submoduleC 9662270...74dc49a:
─────────────────────────────
```

Expected delta output:
```
Submodule submoduleA 9caac0f..a2879d8 (rewind):
─────────────────────────────────
  < Submodule A extra change 2
  < Submodule A extra change 1
  < Submodule A initial commit 4
  < Submodule A initial commit 3

Submodule submoduleB 70b6f0d..955ce29:
─────────────────────────────
  > Submodule B stage change 3
  > Submodule B stage change 2
  > Submodule B stage change 1

Submodule submoduleC 9662270...74dc49a:
─────────────────────────────
  < Submodule C extra change 2
  > Submodule C stage change 3
  < Submodule C extra change 1
  > Submodule C stage change 2
  > Submodule C stage change 1
```

## `diff.submodule=diff`

I don't use this case as often, but it currently seems reasonably well handled to me, likely due to the syntax effectively handling submodule contents with relative paths in the same way as a subdirectory. This convenience also appears to be what leads to the ambiguity in parsing for the `diff.submodule=short` case, so perhaps that's the tradeoff.

# Reproducing example

Here is a rough script (start of a test case?) to generate a structure to highlight each of the cases, and callouts embedded within for where the cases are available to be highlighted.

```
REPO_ROOT=/tmp/submodule-diff-example
REPO_DATE_FMT=%H:%M:%S.%N
mkdir ${REPO_ROOT} && git -C ${REPO_ROOT} init && cd ${REPO_ROOT}
date +${REPO_DATE_FMT} >> baserepo && git add baserepo && git commit -m "Base repo commit 1"

for sub in A B C; do
    git init submodule${sub}
    for iter in $(seq 1 4); do
        date +${REPO_DATE_FMT} >> submodule${sub}/subcontent
        git -C submodule${sub} add subcontent && git -C submodule${sub} commit -m "Submodule ${sub} initial commit $iter"
    done
    # Add initial submodule, message of "Submodule submoduleX 0000000...xxxxxxx (new submodule)", no individual commits
    git submodule add ../bogus-url-${sub} submodule${sub} && \
        git commit -m "Add submodule${sub}" # the diff handling for this is correct in delta

    # Create additional submodule commits
    for iter in $(seq 1 2); do
        date +${REPO_DATE_FMT} >> submodule${sub}/subcontent
        git -C submodule${sub} add subcontent && git -C submodule${sub} commit -m "Submodule ${sub} extra change ${iter}"
    done
    git add submodule${sub} && git commit -m "Update submodule${sub}"
done

git -C submoduleA reset --hard HEAD~4
git -C submoduleC reset --hard HEAD~2

for sub in B C; do
    for iter in $(seq 1 3); do
        date +${REPO_DATE_FMT} >> submodule${sub}/subcontent
        git -C submodule${sub} add subcontent && git -C submodule${sub} commit -m "Submodule ${sub} stage change ${iter}"
    done
done

# Add all submodule updates in single commit to test multiple submodule updates in single commit
git add submodule[A-C] && git commit -m "Update all submodules"

# submoduleA end state is only removed commits
# submoduleB end state is only added commits
# submoduleC is a mixture of removed and added commits (e.g. different branch)

# Manual, inspect superproject history via:
#    "git -c diff.submodule=short log -p | delta --no-gitconfig"
#    "git -c diff.submodule=log log -p | delta --no-gitconfig"
#    "git -c diff.submodule=diff log -p | delta --no-gitconfig"
```



Discussion:
I originally thought I'd also seen cases with `git-status` where submodule details weren't provided as expected, but I wasn't able to identify an occurrence of that while generating my reproducer.
@mrjoel Thanks so much for this fantastically useful write-up (and sorry I didn't respond earlier). I've committed the script to the collection of scripts in `etc/examples` and used it to develop a fix and regression test (#662) for the `diff.submodule = log` case. As you say, this is the most egregious, since delta should never suppress output from git unless it is doing so intentionally. So that now looks like this:

<table><tr><td><img width=300px src="https://user-images.githubusercontent.com/52205/130360159-5314b9b6-5b58-43cb-b773-a60c42af350c.png" alt="image" /></td></tr></table>

I like your suggestion for `diff.submodule = short`. It is slightly more involved to implement, since it requires remembering state over the two successive lines, but perfectly doable. One thing thats making me hesitate is that (correct me if I'm wrong) git's output here seems to be consistent with both (a) a genuine submodule change and (b) a genuine diff, i.e. a user who has written the words "Subproject commit " at the beginning of a line in a file in their repo:

```
diff --git a/submoduleA b/submoduleA
index f4f55af..310b551 160000
--- a/submoduleA
+++ b/submoduleA
@@ -1 +1 @@
-Subproject commit f4f55af749bad858b88976b1c46738056769f2db
+Subproject commit 310b5519561f316541d527e5b0577e7f544f48dc
```

So my hesitation was over adding special-case logic based on words rather than diff syntax.

> might it make sense to consider passing through lines not matching any delta pattern handling as a fallback for other similar cases?

Indeed, delta does try to do this. It was suppressing the submodule log content because it was inappropriately trying to re-use a state in the state machine for cases that needed distinct handling.
> One thing thats making me hesitate is that (correct me if I'm wrong) git's output here seems to be consistent with both (a) a genuine submodule change and (b) a genuine diff, i.e. a user who has written the words "Subproject commit " at the beginning of a line in a file in their repo:

This is indeed the case as far as I've been able to tell, thus my comment of ambiguity above.

The best option I can see is simply a heuristic of diff containing only the first changed line matching that pattern. That wouldn't then generate the summary header if additional lines changed later, but does require state tracking as you mention. That still wouldn't even cover the case of a plain file which happened to contain that matching text, but I'd argue that is such an unlikely corner case to worry about.

Edit: I also don't know how or whether localization would play into that, which would complicate recognition.
> The best option I can see is simply a heuristic of diff containing only the first changed line matching that pattern. 

> That still wouldn't even cover the case of a plain file which happened to contain that matching text, but I'd argue that is such an unlikely corner case to worry about.

Yes, I agree. Thanks -- I've implemented your suggestion in #700.  Instead of the 3 dots I've used a right arrow character with spaces. The main reason I did that is that double-click on `70b6f0d..955ce29` in a terminal emulator will often select the entire thing rather than selecting an individual commit hash. So, it looks like this:

<table><tr><td><img width=600px src="https://user-images.githubusercontent.com/52205/130397143-61f311ba-d3bb-45f2-8d9d-8571bc6275aa.png" alt="image" /></td></tr></table>
> double-click on 70b6f0d..955ce29 in a terminal emulator will often select the entire thing rather than selecting an individual commit hash

That's often an advantage and desirable for directly pasting as a [dotted range notation](https://www.git-scm.com/docs/gitrevisions#_dotted_range_notations), but it definitely depends on one's usage patterns and preferred workflow.

Thanks for the updates!
OK, I've switched that to the two-dot notation as you suggest. I had been thinking of copying the commit hash and doing `git log` but the range notation does seem appropriate. 2b8a894489c00f426d9d29d0c18f749d2508c8e6



Patch:
diff --git a/etc/examples/662-submodules b/etc/examples/662-submodules
new file mode 100755
index 000000000..1f4a51897
--- /dev/null
+++ b/etc/examples/662-submodules
@@ -0,0 +1,45 @@
+#!/bin/bash
+REPO_ROOT=/tmp/submodule-diff-example
+REPO_DATE_FMT=%H:%M:%S.%N
+mkdir ${REPO_ROOT} && git -C ${REPO_ROOT} init && cd ${REPO_ROOT}
+date +${REPO_DATE_FMT} >> baserepo && git add baserepo && git commit -m "Base repo commit 1"
+
+for sub in A B C; do
+    git init submodule${sub}
+    for iter in $(seq 1 4); do
+        date +${REPO_DATE_FMT} >> submodule${sub}/subcontent
+        git -C submodule${sub} add subcontent && git -C submodule${sub} commit -m "Submodule ${sub} initial commit $iter"
+    done
+    # Add initial submodule, message of "Submodule submoduleX 0000000...xxxxxxx (new submodule)", no individual commits
+    git submodule add ../bogus-url-${sub} submodule${sub} && \
+        git commit -m "Add submodule${sub}" # the diff handling for this is correct in delta
+
+    # Create additional submodule commits
+    for iter in $(seq 1 2); do
+        date +${REPO_DATE_FMT} >> submodule${sub}/subcontent
+        git -C submodule${sub} add subcontent && git -C submodule${sub} commit -m "Submodule ${sub} extra change ${iter}"
+    done
+    git add submodule${sub} && git commit -m "Update submodule${sub}"
+done
+
+git -C submoduleA reset --hard HEAD~4
+git -C submoduleC reset --hard HEAD~2
+
+for sub in B C; do
+    for iter in $(seq 1 3); do
+        date +${REPO_DATE_FMT} >> submodule${sub}/subcontent
+        git -C submodule${sub} add subcontent && git -C submodule${sub} commit -m "Submodule ${sub} stage change ${iter}"
+    done
+done
+
+# Add all submodule updates in single commit to test multiple submodule updates in single commit
+git add submodule[A-C] && git commit -m "Update all submodules"
+
+# submoduleA end state is only removed commits
+# submoduleB end state is only added commits
+# submoduleC is a mixture of removed and added commits (e.g. different branch)
+
+# Manual, inspect superproject history via:
+#    "git -c diff.submodule=short log -p | delta --no-gitconfig"
+#    "git -c diff.submodule=log log -p | delta --no-gitconfig"
+#    "git -c diff.submodule=diff log -p | delta --no-gitconfig"
diff --git a/src/config.rs b/src/config.rs
index b0a8cb3c9..acf7d0c99 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -100,6 +100,7 @@ impl Config {
             State::CommitMeta => &self.commit_style,
             State::FileMeta => &self.file_style,
             State::HunkHeader => &self.hunk_header_style,
+            State::Submodule => &self.file_style,
             _ => delta_unreachable("Unreachable code reached in get_style."),
         }
     }
diff --git a/src/delta.rs b/src/delta.rs
index e1770b7ea..c42ba9f11 100644
--- a/src/delta.rs
+++ b/src/delta.rs
@@ -24,6 +24,7 @@ pub enum State {
     HunkZero, // In hunk; unchanged line
     HunkMinus(Option<String>), // In hunk; removed line (raw_line)
     HunkPlus(Option<String>), // In hunk; added line (raw_line)
+    Submodule,
     Unknown,
 }
 
@@ -140,10 +141,11 @@ impl<'a> StateMachine<'a> {
             } else if line.starts_with("@@") {
                 self.handle_hunk_header_line()?
             } else if self.source == Source::DiffUnified && line.starts_with("Only in ")
-                || line.starts_with("Submodule ")
                 || line.starts_with("Binary files ")
             {
-                self.handle_additional_file_meta_cases()?
+                self.handle_additional_cases(State::FileMeta)?
+            } else if line.starts_with("Submodule ") {
+                self.handle_additional_cases(State::Submodule)?
             } else if self.state.is_in_hunk() {
                 // A true hunk line should start with one of: '+', '-', ' '. However, handle_hunk_line
                 // handles all lines until the state transitions away from the hunk states.
@@ -381,10 +383,10 @@ impl<'a> StateMachine<'a> {
         _write_generic_file_meta_header_line(&line, &line, &mut self.painter, self.config)
     }
 
-    fn handle_additional_file_meta_cases(&mut self) -> std::io::Result<bool> {
+    fn handle_additional_cases(&mut self, to_state: State) -> std::io::Result<bool> {
         let mut handled_line = false;
 
-        // Additional FileMeta cases:
+        // Additional cases:
         //
         // 1. When comparing directories with diff -u, if filenames match between the
         //    directories, the files themselves will be compared. However, if an equivalent
@@ -398,7 +400,7 @@ impl<'a> StateMachine<'a> {
         // proposal for more robust parsing logic.
 
         self.painter.paint_buffered_minus_and_plus_lines();
-        self.state = State::FileMeta;
+        self.state = to_state;
         if self.should_handle() {
             self.painter.emit()?;
             _write_generic_file_meta_header_line(


Test patch:
diff --git a/src/tests/test_example_diffs.rs b/src/tests/test_example_diffs.rs
index 3f28d2000..3b249eff0 100644
--- a/src/tests/test_example_diffs.rs
+++ b/src/tests/test_example_diffs.rs
@@ -200,6 +200,16 @@ mod tests {
         assert_eq!(strip_ansi_codes(&output), DIFF_WITH_MERGE_CONFLICT);
     }
 
+    #[test]
+    fn test_submodule_diff_log() {
+        // See etc/examples/662-submodules
+        // diff.submodule = log
+        let config = integration_test_utils::make_config_from_args(&["--width", "49"]);
+        let output = integration_test_utils::run_delta(SUBMODULE_DIFF_LOG, &config);
+        let output = strip_ansi_codes(&output);
+        assert_eq!(output, SUBMODULE_DIFF_LOG_EXPECTED_OUTPUT);
+    }
+
     #[test]
     fn test_submodule_contains_untracked_content() {
         let config = integration_test_utils::make_config_from_args(&[]);
@@ -1902,6 +1912,62 @@ This is a regular file that contains:
 +Some text with a plus
 ";
 
+    // See etc/examples/662-submodules
+    // diff.submodule = log
+    const SUBMODULE_DIFF_LOG: &str = "\
+commit ccb444baa861fdcb14d411b471a74614ed28776d
+Author: Dan Davison <dandavison7@gmail.com>
+Date:   Sat Aug 21 18:56:34 2021 -0700
+
+    Update all submodules
+
+Submodule submoduleA f4f55af..310b551 (rewind):
+  < Submodule A extra change 2
+  < Submodule A extra change 1
+  < Submodule A initial commit 4
+  < Submodule A initial commit 3
+Submodule submoduleB 0ffa700..0c8b00d:
+  > Submodule B stage change 3
+  > Submodule B stage change 2
+  > Submodule B stage change 1
+Submodule submoduleC 9f3b744...e04f848:
+  > Submodule C stage change 3
+  > Submodule C stage change 2
+  < Submodule C extra change 2
+  > Submodule C stage change 1
+  < Submodule C extra change 1
+";
+
+    const SUBMODULE_DIFF_LOG_EXPECTED_OUTPUT: &str = "\
+commit ccb444baa861fdcb14d411b471a74614ed28776d
+Author: Dan Davison <dandavison7@gmail.com>
+Date:   Sat Aug 21 18:56:34 2021 -0700
+
+    Update all submodules
+
+
+Submodule submoduleA f4f55af..310b551 (rewind):
+─────────────────────────────────────────────────
+  < Submodule A extra change 2
+  < Submodule A extra change 1
+  < Submodule A initial commit 4
+  < Submodule A initial commit 3
+
+Submodule submoduleB 0ffa700..0c8b00d:
+─────────────────────────────────────────────────
+  > Submodule B stage change 3
+  > Submodule B stage change 2
+  > Submodule B stage change 1
+
+Submodule submoduleC 9f3b744...e04f848:
+─────────────────────────────────────────────────
+  > Submodule C stage change 3
+  > Submodule C stage change 2
+  < Submodule C extra change 2
+  > Submodule C stage change 1
+  < Submodule C extra change 1
+";
+
     const SUBMODULE_CONTAINS_UNTRACKED_CONTENT_INPUT: &str = "\
 --- a
 +++ b