Repository: sharkdp/bat (language: rust)

Issue:
Make bat autodetect CSV separator
Hi,

CSV files often have a semicolon `;` as separator instead of a simple comma `,`. There also exist so-called TSV files where the separator is a tabulation. Would it be possible for `bat` to autodetect that ?


Discussion / hints:
Thank you for your request. That would be great, I agree.

I'm not sure if we can really "detect" the separator, but I guess we could write a syntax file that would accept all `,`, `;` and `\t`. @keith-hall could certainly tell us more about whether this could work or not.

Another thing that I have been thinking about is to use a concept similar to https://packagecontrol.io/packages/rainbow_csv where we would display different CSV columns with different (or alternating) colors. Unfortunately, https://packagecontrol.io/packages/rainbow_csv does not seem to properly support quoting (such as `1,"column two with text, and a comma",3,…`.
> we could write a syntax file that would accept all `,`, `;` and `\t`

Yep, that's certainly possible 

> Unfortunately, https://packagecontrol.io/packages/rainbow_csv does not seem to properly support quoting (such as `1,"column two with text, and a comma",3,…`.

I'm surprised at that, the screenshot in the readme of that package seems to suggest it handles quoted values. 
> I'm surprised at that, the screenshot in the readme of that package seems to suggest it handles quoted values.

My mistake. It does. I only looked at the last lines of our test file previously:

![image](https://user-images.githubusercontent.com/4209276/110248552-10377000-7f72-11eb-9441-711efcd44938.png)

but that example is pretty challenging, if not to say... broken.

The "rainbow" doesn't look too nice though. Not sure if we would have to make any changes to `bat` theming to support that properly.
> but that example is pretty challenging, if not to say... broken.

I guess some CSV parsers support newlines in quoted values (i.e. so it's not treated as a row-terminator), and double quotes to escape the quotation marks, so it could be considered not broken, just unusual - one shouldn't really use CSV in such cases IMHO ;)

> The "rainbow" doesn't look too nice though. Not sure if we would have to make any changes to `bat` theming to support that properly.

Can you be more specific about what you don't like about it please? Does it look different in `bat` versus Sublime Text? I guess it's also worth comparing it to what we have now - our current CSV highlighting highlights numbers, commas and quotes if I understand correctly, and everything else also gets a color instead of the default foreground color. (This latter part makes it very noisy IMO.)

> Can you be more specific about what you don't like about it please?

Mostly the colors. But that obviously depends on the theme. I don't like that the default (white) color is used. And that the dim comment color is used ("city"). Also, it would  be great if the separator character would be highlighted in a different (consistent) way.
> I don't like that the default (white) color is used. And that the dim comment color is used ("city"). Also, it would be great if the separator character would be highlighted in a different (consistent) way.

I completely agree :)

I've started working on this, and it colors commas and semi-colons as separators, and treats tabs as separators but doesn't currently color them... Here is how it looks so far:
![image](https://user-images.githubusercontent.com/11882719/112734322-b234f180-8f4d-11eb-9512-5a869622833c.png)

What do we think about quoted strings? Currently, I've got strings as their own separate color, i.e. the same behavior as other syntaxes, and escaped quotes are highlighted the same as character escapes are usually in quoted strings. But do we want to keep the "Advanced CSV" behavior of each field gets a separate color, regardless of whether it is quoted or not?

Monokai only really has about 5 different colors to choose from, when we ignore strings and comment colors, so that's all I have added support for, at least for now. Feedback welcome, after which I can tidy it up and submit a PR :) (Would it make sense to leave the default (white in this case) color for separators or anything to give us an extra color to work with?)

Fix patch:
diff --git a/.gitmodules b/.gitmodules
index 4b6bc34ee8..12beb70cd7 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -61,9 +61,6 @@
 [submodule "assets/syntaxes/Puppet"]
 	path = assets/syntaxes/02_Extra/Puppet
 	url = https://github.com/russCloak/SublimePuppet
-[submodule "assets/syntaxes/CSV"]
-	path = assets/syntaxes/02_Extra/CSV
-	url = https://github.com/wadetb/Sublime-Text-Advanced-CSV
 [submodule "assets/themes/onehalf"]
 	path = assets/themes/onehalf
 	url = https://github.com/sonph/onehalf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2b02eb414d..357b22731f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,8 @@
 
 ## Syntaxes
 
+- Replaced "Advanced CSV" with a custom CSV syntax definition written especially for `bat`; see #1574 (@keith-hall)
+
 ## New themes
 
 ## `bat` as a library
diff --git a/assets/patches/Monokai-Extended.tmTheme.patch b/assets/patches/Monokai-Extended.tmTheme.patch
index 24573a7b31..16624d5fbe 100644
--- a/assets/patches/Monokai-Extended.tmTheme.patch
+++ b/assets/patches/Monokai-Extended.tmTheme.patch
@@ -7,7 +7,7 @@ index 9c2aa3e..180cbbf 100644
        <string>JSON String</string>
        <key>scope</key>
 -      <string>meta.structure.dictionary.json string.quoted.double.json</string>
-+      <string>meta.mapping.key.json string.quoted.double.json</string>
++      <string>meta.mapping.key.json string.quoted.double.json, punctuation.separator.sequence.csv</string>
        <key>settings</key>
        <dict>
          <key>foreground</key>
diff --git a/assets/syntaxes/02_Extra/CSV b/assets/syntaxes/02_Extra/CSV
deleted file mode 160000
index 4786d037a7..0000000000
--- a/assets/syntaxes/02_Extra/CSV
+++ /dev/null
@@ -1,1 +0,0 @@
-Subproject commit 4786d037a761bc45e516f6b0624a839919ec6d05
diff --git a/assets/syntaxes/02_Extra/CSV.sublime-syntax b/assets/syntaxes/02_Extra/CSV.sublime-syntax
index 1bbc5b9763..cca7cd2ca9 100644
--- a/assets/syntaxes/02_Extra/CSV.sublime-syntax
+++ b/assets/syntaxes/02_Extra/CSV.sublime-syntax
@@ -1,46 +1,78 @@
 %YAML 1.2
 ---
-# http://www.sublimetext.com/docs/3/syntax.html
-name: Advanced CSV
+# See http://www.sublimetext.com/docs/3/syntax.html
+name: Comma Separated Values
 file_extensions:
   - csv
   - tsv
-scope: text.advanced_csv
+scope: text.csv
+variables:
+  field_separator: (?:[,;\t])
+  record_separator: (?:$\n?)
 contexts:
-  main:
-    - match: (\")
-      captures:
-        1: string.quoted.double.advanced_csv
-      push:
-        - meta_scope: meta.quoted.advanced_csv
-        - match: (\")
-          captures:
-            1: string.quoted.double.advanced_csv
-          pop: true
-        - include: main
-    - match: '(\[([+-]?\d*)(\:)?([+-]?\d*)(\,)?([+-]?\d*)(\:)?([+-]?\d*)\])?\s*([<>v^])?\s*(=)'
-      captures:
-        1: keyword.operator.advanced_csv
-        2: constant.numeric.formula.advanced_csv
-        4: constant.numeric.formula.advanced_csv
-        6: constant.numeric.formula.advanced_csv
-        8: constant.numeric.formula.advanced_csv
-        9: keyword.operator.advanced_csv
-        10: keyword.operator.advanced_csv
+  prototype:
+    - match: (?={{record_separator}})
+      pop: true
+  fields:
+    - match: ''
       push:
-        - meta_scope: meta.range.advanced_csv
-        - match: (?=(\")|$)
-          pop: true
-        - include: scope:source.python
-    - match: '(?<=^|,|\s|\")([0-9.eE+-]+)(?=$|,|\s|\")'
-      scope: meta.number.advanced_csv
-      captures:
-        1: constant.numeric.advanced_csv
-    - match: '(?<=^|,|\s|\")([^, \t\"]+)(?=$|,|\s|\")'
-      scope: meta.nonnumber.advanced_csv
-      captures:
-        1: storage.type.advanced_csv
-    - match: (\,)
-      scope: meta.delimiter.advanced_csv
-      captures:
-        1: keyword.operator.advanced_csv
+        - field_or_record_separator
+        - field4
+        - field_or_record_separator
+        - field3
+        - field_or_record_separator
+        - field2
+        - field_or_record_separator
+        - field1
+  main:
+    - meta_include_prototype: false
+    - match: '^'
+      set: fields
+
+  field_or_record_separator:
+    - meta_include_prototype: false
+    - match: '{{record_separator}}'
+      scope: punctuation.terminator.record.csv
+      pop: true
+    - match: '{{field_separator}}'
+      scope: punctuation.separator.sequence.csv
+      pop: true
+
+  field_contents:
+    - match: '"'
+      scope: punctuation.definition.string.begin.csv
+      push: double_quoted_string
+
+    - match: (?={{field_separator}}|{{record_separator}})
+      pop: true
+
+  double_quoted_string:
+    - meta_include_prototype: false
+    - meta_scope: string.quoted.double.csv
+    - match: '""'
+      scope: constant.character.escape.csv
+    - match: '"'
+      scope: punctuation.definition.string.end.csv
+      pop: true
+
+  field1:
+    - match: ''
+      set:
+        - meta_content_scope: meta.field-1.csv support.type
+        - include: field_contents
+  field2:
+    - match: ''
+      set:
+        - meta_content_scope: meta.field-2.csv support.function
+        - include: field_contents
+  field3:
+    - match: ''
+      set:
+        - meta_content_scope: meta.field-3.csv constant.numeric
+        - include: field_contents
+  field4:
+    - match: ''
+      set:
+        - meta_content_scope: meta.field-4.csv keyword.operator
+        - include: field_contents
+


Test patch:
diff --git a/assets/syntaxes/02_Extra/syntax_test_csv.csv b/assets/syntaxes/02_Extra/syntax_test_csv.csv
new file mode 100644
index 0000000000..c997b2bd4a
--- /dev/null
+++ b/assets/syntaxes/02_Extra/syntax_test_csv.csv
@@ -0,0 +1,60 @@
+# SYNTAX TEST "CSV.sublime-syntax"
+Forename,Surname,House No.,Street,City,Postcode
+#^^^^^^^ meta.field-1
+#       ^ punctuation.separator.sequence
+#        ^^^^^^^ meta.field-2
+#               ^ punctuation.separator.sequence
+#                ^^^^^^^^^ meta.field-3
+#                         ^ punctuation.separator.sequence
+#                          ^^^^^^ meta.field-4
+#                                ^ punctuation.separator.sequence
+#                                 ^^^^ meta.field-1
+#                                     ^ punctuation.separator.sequence
+#                                      ^^^^^^^^ meta.field-2
+John,Doe,87,Logon Road,Citiville,TU43 6GH
+#^^^ meta.field-1
+#   ^ punctuation.separator.sequence
+#    ^^^ meta.field-2
+#       ^ punctuation.separator.sequence
+#        ^^ meta.field-3
+#          ^ punctuation.separator.sequence
+#           ^^^^^^^^^^ meta.field-4
+#                     ^ punctuation.separator.sequence
+#                      ^^^^^^^^^ meta.field-1
+#                               ^ punctuation.separator.sequence
+#                                ^^^^^^^^ meta.field-2
+Jane,Doe,22,"""Fun"" Street","The City, County",FA90 1XA
+#^^^ meta.field-1
+#   ^ punctuation.separator.sequence
+#    ^^^ meta.field-2
+#       ^ punctuation.separator.sequence
+#        ^^ meta.field-3
+#          ^ punctuation.separator.sequence
+#           ^^^^^^^^^^^^^^^^ meta.field-4 string.quoted.double
+#            ^^ constant.character.escape
+#                 ^^ constant.character.escape
+#                           ^ punctuation.separator.sequence
+#                            ^ meta.field-1 string.quoted.double punctuation.definition.string.begin
+#                             ^^^^^^^^^^^^^^^^^ meta.field-1 string.quoted.double
+#                                             ^ meta.field-1 string.quoted.double punctuation.definition.string.end
+#                                              ^ punctuation.separator.sequence
+#                                               ^^^^^^^^ meta.field-2
+Fred,Bloggs,"19","Street
+#^^^ meta.field-1
+#   ^ punctuation.separator.sequence
+#    ^^^^^^ meta.field-2
+#          ^ punctuation.separator.sequence
+#            ^^^ meta.field-3 string.quoted.double
+#               ^ punctuation.separator.sequence
+#                ^ meta.field-4 punctuation.definition.string.begin
+#                ^^^^^^^^ meta.field-4 string.quoted.double
+Spanning Multiple ""
+#^^^^^^^^^^^^^^^^^^^^ meta.field-4 string.quoted.double
+#                 ^^ constant.character.escape
+Lines",Citee,P0stc0d3
+#^^^^^ meta.field-4 string.quoted.double
+#    ^ meta.field-4 punctuation.definition.string.end
+#     ^ punctuation.separator.sequence
+#      ^^^^^ meta.field-1
+#           ^ punctuation.separator.sequence
+#            ^^^^^^^^ meta.field-2
diff --git a/assets/syntaxes/02_Extra/syntax_test_tsv.tsv b/assets/syntaxes/02_Extra/syntax_test_tsv.tsv
new file mode 100644
index 0000000000..8c2645b8d3
--- /dev/null
+++ b/assets/syntaxes/02_Extra/syntax_test_tsv.tsv
@@ -0,0 +1,60 @@
+# SYNTAX TEST "CSV.sublime-syntax"
+Forename	Surname	House No.	Street	City	Postcode
+#^^^^^^^ meta.field-1
+#       ^ punctuation.separator.sequence
+#        ^^^^^^^ meta.field-2
+#               ^ punctuation.separator.sequence
+#                ^^^^^^^^^ meta.field-3
+#                         ^ punctuation.separator.sequence
+#                          ^^^^^^ meta.field-4
+#                                ^ punctuation.separator.sequence
+#                                 ^^^^ meta.field-1
+#                                     ^ punctuation.separator.sequence
+#                                      ^^^^^^^^ meta.field-2
+John	Doe	87	Logon Road	Citiville	TU43 6GH
+#^^^ meta.field-1
+#   ^ punctuation.separator.sequence
+#    ^^^ meta.field-2
+#       ^ punctuation.separator.sequence
+#        ^^ meta.field-3
+#          ^ punctuation.separator.sequence
+#           ^^^^^^^^^^ meta.field-4
+#                     ^ punctuation.separator.sequence
+#                      ^^^^^^^^^ meta.field-1
+#                               ^ punctuation.separator.sequence
+#                                ^^^^^^^^ meta.field-2
+Jane	Doe	22	"""Fun"" Street"	"The City	 County"	FA90 1XA
+#^^^ meta.field-1
+#   ^ punctuation.separator.sequence
+#    ^^^ meta.field-2
+#       ^ punctuation.separator.sequence
+#        ^^ meta.field-3
+#          ^ punctuation.separator.sequence
+#           ^^^^^^^^^^^^^^^^ meta.field-4 string.quoted.double
+#            ^^ constant.character.escape
+#                 ^^ constant.character.escape
+#                           ^ punctuation.separator.sequence
+#                            ^ meta.field-1 string.quoted.double punctuation.definition.string.begin
+#                             ^^^^^^^^^^^^^^^^^ meta.field-1 string.quoted.double
+#                                             ^ meta.field-1 string.quoted.double punctuation.definition.string.end
+#                                              ^ punctuation.separator.sequence
+#                                               ^^^^^^^^ meta.field-2
+Fred	Bloggs	"19"	"Street
+#^^^ meta.field-1
+#   ^ punctuation.separator.sequence
+#    ^^^^^^ meta.field-2
+#          ^ punctuation.separator.sequence
+#            ^^^ meta.field-3 string.quoted.double
+#               ^ punctuation.separator.sequence
+#                ^ meta.field-4 punctuation.definition.string.begin
+#                ^^^^^^^^ meta.field-4 string.quoted.double
+Spanning Multiple ""
+#^^^^^^^^^^^^^^^^^^^^ meta.field-4 string.quoted.double
+#                 ^^ constant.character.escape
+Lines"	Citee	P0stc0d3
+#^^^^^ meta.field-4 string.quoted.double
+#    ^ meta.field-4 punctuation.definition.string.end
+#     ^ punctuation.separator.sequence
+#      ^^^^^ meta.field-1
+#           ^ punctuation.separator.sequence
+#            ^^^^^^^^ meta.field-2
diff --git a/tests/syntax-tests/highlighted/CSV/comma_in_quotes.csv b/tests/syntax-tests/highlighted/CSV/comma_in_quotes.csv
index 2ad5b91470..9bd0da1a6c 100644
--- a/tests/syntax-tests/highlighted/CSV/comma_in_quotes.csv
+++ b/tests/syntax-tests/highlighted/CSV/comma_in_quotes.csv
@@ -1,7 +1,7 @@
-[3;38;2;102;217;239mfirst[0m[38;2;249;38;114m,[0m[3;38;2;102;217;239mlast[0m[38;2;249;38;114m,[0m[3;38;2;102;217;239maddress[0m[38;2;249;38;114m,[0m[3;38;2;102;217;239mcity[0m[38;2;249;38;114m,[0m[3;38;2;102;217;239mzip[0m
-[3;38;2;102;217;239mJohn[0m[38;2;249;38;114m,[0m[3;38;2;102;217;239mDoe[0m[38;2;249;38;114m,[0m[38;2;190;132;255m120[0m[38;2;248;248;242m [0m[3;38;2;102;217;239many[0m[38;2;248;248;242m [0m[3;38;2;102;217;239mst.[0m[38;2;249;38;114m,[0m[38;2;230;219;116m"[0m[3;38;2;102;217;239mAnytown[0m[38;2;249;38;114m,[0m[38;2;248;248;242m [0m[3;38;2;102;217;239mWW[0m[38;2;230;219;116m"[0m[38;2;249;38;114m,[0m[38;2;190;132;255m08123[0m
-[3;38;2;102;217;239ma[0m[38;2;249;38;114m,[0m[3;38;2;102;217;239mb[0m
-[38;2;190;132;255m1[0m[38;2;249;38;114m,[0m[38;2;230;219;116m"[0m[3;38;2;102;217;239mha[0m[38;2;248;248;242m [0m
-[38;2;230;219;116m"[0m[38;2;230;219;116m"[0m[3;38;2;102;217;239mha[0m[38;2;230;219;116m"[0m[38;2;230;219;116m"[0m[38;2;248;248;242m [0m
-[3;38;2;102;217;239mha[0m[38;2;230;219;116m"[0m[38;2;249;38;114m,[0m[38;2;190;132;255m120[0m[38;2;248;248;242m [0m[3;38;2;102;217;239many[0m[38;2;248;248;242m [0m[3;38;2;102;217;239mst.[0m[38;2;249;38;114m,[0m[38;2;230;219;116m"[0m[3;38;2;102;217;239mAnytown[0m[38;2;249;38;114m,[0m[38;2;248;248;242m [0m[3;38;2;102;217;239mWW[0m[38;2;230;219;116m"[0m[38;2;249;38;114m,[0m[38;2;190;132;255m08123[0m
-[38;2;190;132;255m3[0m[38;2;249;38;114m,[0m[38;2;190;132;255m4[0m[38;2;249;38;114m,[0m[38;2;190;132;255m120[0m[38;2;248;248;242m [0m[3;38;2;102;217;239many[0m[38;2;248;248;242m [0m[3;38;2;102;217;239mst.[0m[38;2;249;38;114m,[0m[38;2;230;219;116m"[0m[3;38;2;102;217;239mAnytown[0m[38;2;249;38;114m,[0m[38;2;248;248;242m [0m[3;38;2;102;217;239mWW[0m[38;2;230;219;116m"[0m[38;2;249;38;114m,[0m[38;2;190;132;255m08123[0m
+[3;38;2;166;226;46mfirst[0m[38;2;253;151;31m,[0m[38;2;102;217;239mlast[0m[38;2;253;151;31m,[0m[38;2;190;132;255maddress[0m[38;2;253;151;31m,[0m[38;2;249;38;114mcity[0m[38;2;253;151;31m,[0m[3;38;2;166;226;46mzip[0m
+[3;38;2;166;226;46mJohn[0m[38;2;253;151;31m,[0m[38;2;102;217;239mDoe[0m[38;2;253;151;31m,[0m[38;2;190;132;255m120 any st.[0m[38;2;253;151;31m,[0m[38;2;230;219;116m"[0m[38;2;230;219;116mAnytown, WW[0m[38;2;230;219;116m"[0m[38;2;253;151;31m,[0m[3;38;2;166;226;46m08123[0m
+[3;38;2;166;226;46ma[0m[38;2;253;151;31m,[0m[38;2;102;217;239mb[0m
+[3;38;2;166;226;46m1[0m[38;2;253;151;31m,[0m[38;2;230;219;116m"[0m[38;2;230;219;116mha [0m
+[38;2;190;132;255m""[0m[38;2;230;219;116mha[0m[38;2;190;132;255m""[0m[38;2;230;219;116m [0m
+[38;2;230;219;116mha[0m[38;2;230;219;116m"[0m[38;2;253;151;31m,[0m[38;2;190;132;255m120 any st.[0m[38;2;253;151;31m,[0m[38;2;230;219;116m"[0m[38;2;230;219;116mAnytown, WW[0m[38;2;230;219;116m"[0m[38;2;253;151;31m,[0m[3;38;2;166;226;46m08123[0m
+[3;38;2;166;226;46m3[0m[38;2;253;151;31m,[0m[38;2;102;217;239m4[0m[38;2;253;151;31m,[0m[38;2;190;132;255m120 any st.[0m[38;2;253;151;31m,[0m[38;2;230;219;116m"[0m[38;2;230;219;116mAnytown, WW[0m[38;2;230;219;116m"[0m[38;2;253;151;31m,[0m[3;38;2;166;226;46m08123[0m