Repository: alanmcruickshank/sqlfluff

Issue:
FileSegment error raised when using tabs
Hey,
I got the following error:

TypeError: In constructing <class 'sqlfluff.parser.segments_file.FileSegment'>, found an element of the segments tuple which isn't contigious with previous: <whitespace_RawSegment: ([4](1, 2, 2)) '\t'> > <newline_RawSegment: ([4](1, 2, 2)) '\n'>. End pos: [5](1, 2, 3). Prev String: '\t'

By using this input file 
/*
	
*/

Please note that in the comment is an invisible tab in the otherwise empty line.


Discussion:

Patch:
diff --git a/src/sqlfluff/parser/lexer.py b/src/sqlfluff/parser/lexer.py
index 1eb4da37..ebbcb859 100644
--- a/src/sqlfluff/parser/lexer.py
+++ b/src/sqlfluff/parser/lexer.py
@@ -69,6 +69,10 @@ class SingletonMatcher:
                     )
                     idx = trim_span[1]
                     cont_pos_buff = cont_pos_buff.advance_by(matched[:trim_span[1]])
+                    # Have we consumed the whole string? This avoids us having
+                    # an empty string on the end.
+                    if idx == len(matched):
+                        break
                 # Is it at the end?
                 if trim_span[1] == len(matched):
                     seg_buff += (


Test patch:
diff --git a/test/parser_lexer_test.py b/test/parser_lexer_test.py
index 6b8ba2a9..8fbd5301 100644
--- a/test/parser_lexer_test.py
+++ b/test/parser_lexer_test.py
@@ -46,8 +46,9 @@ def assert_matches(instring, matcher, matchstring):
         ("abc -- comment \nblah", ['abc', ' ', "-- comment ", "\n", "blah"]),
         ("abc # comment \nblah", ['abc', ' ', "# comment ", "\n", "blah"]),
         # Note the more complicated parsing of block comments.
-        # This tests subdivision and trimming
+        # This tests subdivision and trimming (incl the empty case)
         ("abc /* comment \nblah*/", ['abc', ' ', "/* comment", " ", "\n", "blah*/"]),
+        ("abc /*\n\t\n*/", ['abc', ' ', "/*", "\n", "\t", "\n", "*/"]),
         # Test Singletons
         ("*-+bd/", ['*', '-', '+', 'bd', '/']),
         # Test Negatives and Minus