Repository: mholt/PapaParse (JavaScript)

Issue:
Error parsing file when CR appears in header in quotes
Hi Matt, thanks for the awesome library!

I have encountered a potential issue regarding newline characters, and it would be great to hear your thoughts on it. 

Some files I'm using rather unusually have CR characters in the headers, although they appear in quotes. The end of the line is a normal LF character. I think when PapaParse tries to automatically detect the newline characters, it will use CR which leads to parsing errors (MissingQuotes). Obviously I could tell PapaParse the newline character, but ideally I'd like to continue using the automatic detection because I don't always know what it's going to be.

Would it be possible to ignore newline characters which are in quotes when trying to guess the CSV end of line character?

Here is an example file which exhibits the problem: https://www.dropbox.com/s/d6qkvywinjekulh/header-newline.csv?dl=0

Thanks!

Unable to parse files with \n and \r\n
I am having a little issue with PapaParse when parsing a file that contains newlines in fields. In particular, the file has `\r\n` as line separator, but inside fields there are `\n` as new lines.

When I load such files, PapaParse recognizes `\n` as newline delimiter instead of `\r\n`, so the entire file is read in the first row.

An example file follows, base64-encoded (there is BOM at the beginning).

```
77u/Zmlyc3Qtcm93LCJMaW5lICIid2l0aCBxdW90ZWQiIiB0ZXh0CgpGb28gYmFyIGJheiINCnNl
Y29uZC1yb3csInNvbWUgb3RoZXIgIiJxdW90ZWQiIiB0ZXh0IAoKTGluZSBhZnRlciBlbXB0eSIN
Cmxhc3QtZW1wdHktbGluZSw=
```

In particular, when I try parsing the file I get

```
// ff is the file string above
Papa.parse(atob(ff)).data.length // 1
JSON.stringify(Papa.parse(atob(ff)))
"{\"data\":[[\"ï»¿first-row\",\"Line \\\"\\\"with quoted\\\"\\\" text\\n\\nFoo bar baz\\\"\\r\\nsecond-row,\\\"some other \\\"\\\"quoted\\\"\\\" text \\n\\nLine after empty\\\"\\r\\nlast-empty-line,\"]],\"errors\":[{\"type\":\"Quotes\",\"code\":\"MissingQuotes\",\"message\":\"Quoted field unterminated\",\"row\":0,\"index\":14}],\"meta\":{\"delimiter\":\",\",\"linebreak\":\"\\n\",\"aborted\":false,\"truncated\":false,\"cursor\":131}}"
```

I don't know what delimiters are used inside fields and between records, so I am unable to specify the `newline` option when parsing. As a reference, LibreOffice is parsing the above file correctly.

Am I missing some option or is this a bug?
Thanks!


Discussion:
I've added a failing test here if it helps: https://github.com/lboynton/PapaParse/commit/98efe3063fbcd275a074c38d9358674b2e38e399

Hi Lee,

Thanks for opening the issue. I haven't had much time to look at it in detail yet as I'm at a conference today, but I can look into it over the next few weeks to see if I can understand the behavior you're describing. Thanks for providing an example file; I'll look soon.

No problem, thanks for responding so quickly :)

This is how we currently detect line endings: https://github.com/mholt/PapaParse/blob/9183ffb2d815bfe22aca0c65782b8b0eeeb07d75/papaparse.js#L944-L961
1. If there are no `\r` in the first 1 MB of the file, we assume `\n`.
2. Otherwise, we count how many `\n` appear immediately following an `\r`
3. If there are close to the number of `\n`s as `\r`s, assume `\r\n`, otherwise assume `\r`.

So what's happening with your input is that the `\r` is confusing the parser because there are so few `\n` characters (very few rows). But add a few more rows without `\r` and the parser won't get confused. (Try it!)

Is your data really frequently as short as 2-3 rows? If your carriage return characters only appear in the header then the parser should be able to guess correctly with at least a few rows.

Thanks for having a look into my issue Matt. I think my problem though is that my file uses `\n` line endings which in the list above is never an outcome of the line ending detection if there is a `\r` in the header (albeit in quotes)?

My data is rarely that short btw, I was trying to provide the smallest possible test case.

Updated test case: https://github.com/mholt/PapaParse/compare/master...lboynton:newline-problem

I think I see what you mean now. This should get fixed.

I seem to have a similar problem...
some of my quoted string fields may contain either of the newline characters (carriage return or line feed). I was expecting these charcters to be assumed to be part of the string if found within quoted strings, but papaparse is interpretting them as record newlines. I am using auto newlines.

Note that when I originally write the csv files using php, php sees these characters as parts of strings, and automatically encloses these strings in quotes.

I'm having the same issues as @dpatte. Did you ever get this solved?

No, I've been too busy with other projects lately, but pull requests are welcome! In fact, I wonder if #281 would have any effect on this issue.

Or funding work on this project would also be a good way to help get some things done.

I solved it by prescanning my input and converting CR and LF to character strings 
`"<CR>"` and `"<LF>"`
parsing, then restoring them to CR LF after the parse. Worked for me.

Hi All, 

I have tryied to solve the problem by replacing all the text inside quotes by an empty string as it's not relevant for detecting the line delimiter. See https://github.com/mholt/PapaParse/compare/master...newline-problem

But it seems to not solve the test case. Any help will be much appreciated. 

P.D: I refactored a litle bit the test case (See: https://github.com/mholt/PapaParse/commit/a3e3e6dc94201b255a6dd462feea2d09982b3f59) as it was an extra comment and I think that skipEmpty lines is not requied. 

have you tryied using \r\n as separator on config file? 
I'm closing the issue as I understand it has been already solved. Reopen if you don't agree
Sorry for this late answer. No, I don't think issue is solved: as I said, I am unable to determine the newlines used inside the text, so I am not sure if only `\n` are used or `\r\n` appears both in text and as file line separator.

Since fields with newlines are quoted, I expect PapaParse to have higher precedence for them than for newlines to determine when a line is ended. In other words:

    "foo","bar","baz
    qux"
    "one","two","three"

should be parsed correctly independently on the line terminator used. This is not the case as far as I can tell.
Ok, could you please post the config you are using and a sample file wich contains the line break inside the quotes?
I already did provide an example in my original post, is another one required?

Although I did not specify any version: I am using 4.3.0.
same issue



Patch:
diff --git a/papaparse.js b/papaparse.js
index 5f670bd7..23c78896 100755
--- a/papaparse.js
+++ b/papaparse.js
@@ -1008,8 +1008,9 @@
 		 */
 		this.parse = function(input, baseIndex, ignoreLastRow)
 		{
+			var quoteChar = _config.quoteChar || '"';
 			if (!_config.newline)
-				_config.newline = guessLineEndings(input);
+				_config.newline = guessLineEndings(input, quoteChar);
 
 			_delimiterError = false;
 			if (!_config.delimiter)
@@ -1252,9 +1253,12 @@
 			};
 		}
 
-		function guessLineEndings(input)
+		function guessLineEndings(input, quoteChar)
 		{
 			input = input.substr(0, 1024 * 1024);	// max length 1 MB
+			// Replace all the text inside quotes
+			var re = new RegExp(escapeRegExp(quoteChar) + '([^]*?)' + escapeRegExp(quoteChar), 'gm');
+			input = input.replace(re, '');
 
 			var r = input.split('\r');
 
@@ -1286,9 +1290,10 @@
 		}
 	}
 
-
-
-
+	/** https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions */
+	function escapeRegExp(string) {
+		return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
+	}
 
 	/** The core parser implements speedy and correct CSV parsing */
 	function Parser(config)


Test patch:
diff --git a/tests/test-cases.js b/tests/test-cases.js
index ad0d9a58..df5c1524 100644
--- a/tests/test-cases.js
+++ b/tests/test-cases.js
@@ -1208,6 +1208,136 @@ var PARSE_TESTS = [
 			data: [{'a': 'c', 'b': 'd'}],
 			errors: []
 		}
+	},
+	{
+		description: "Carriage return in header inside quotes, with line feed endings",
+		input: '"a\r\na","b"\n"c","d"\n"e","f"\n"g","h"\n"i","j"',
+		config: {},
+		expected: {
+			data: [['a\r\na', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: []
+		}
+	},
+	{
+		description: "Line feed in header inside quotes, with carriage return + line feed endings",
+		input: '"a\na","b"\r\n"c","d"\r\n"e","f"\r\n"g","h"\r\n"i","j"',
+		config: {},
+		expected: {
+			data: [['a\na', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: []
+		}
+	},
+	{
+		description: "Using \\r\\n endings uses \\r\\n linebreak",
+		input: 'a,b\r\nc,d\r\ne,f\r\ng,h\r\ni,j',
+		config: {},
+		expected: {
+			data: [['a', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: [],
+			meta: {
+				linebreak: '\r\n',
+				delimiter: ',',
+				cursor: 23,
+				aborted: false,
+				truncated: false
+			}
+		}
+	},
+	{
+		description: "Using \\n endings uses \\n linebreak",
+		input: 'a,b\nc,d\ne,f\ng,h\ni,j',
+		config: {},
+		expected: {
+			data: [['a', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: [],
+			meta: {
+				linebreak: '\n',
+				delimiter: ',',
+				cursor: 19,
+				aborted: false,
+				truncated: false
+			}
+		}
+	},
+	{
+		description: "Using \\r\\n endings with \\r\\n in header field uses \\r\\n linebreak",
+		input: '"a\r\na",b\r\nc,d\r\ne,f\r\ng,h\r\ni,j',
+		config: {},
+		expected: {
+			data: [['a\r\na', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: [],
+			meta: {
+				linebreak: '\r\n',
+				delimiter: ',',
+				cursor: 28,
+				aborted: false,
+				truncated: false
+			}
+		}
+	},
+	{
+		description: "Using \\r\\n endings with \\n in header field uses \\r\\n linebreak",
+		input: '"a\na",b\r\nc,d\r\ne,f\r\ng,h\r\ni,j',
+		config: {},
+		expected: {
+			data: [['a\na', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: [],
+			meta: {
+				linebreak: '\r\n',
+				delimiter: ',',
+				cursor: 27,
+				aborted: false,
+				truncated: false
+			}
+		}
+	},
+	{
+		description: "Using \\r\\n endings with \\n in header field with skip empty lines uses \\r\\n linebreak",
+		input: '"a\na",b\r\nc,d\r\ne,f\r\ng,h\r\ni,j\r\n',
+		config: {skipEmptyLines: true},
+		expected: {
+			data: [['a\na', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: [],
+			meta: {
+				linebreak: '\r\n',
+				delimiter: ',',
+				cursor: 29,
+				aborted: false,
+				truncated: false
+			}
+		}
+	},
+	{
+		description: "Using \\n endings with \\r\\n in header field uses \\n linebreak",
+		input: '"a\r\na",b\nc,d\ne,f\ng,h\ni,j',
+		config: {},
+		expected: {
+			data: [['a\r\na', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: [],
+			meta: {
+				linebreak: '\n',
+				delimiter: ',',
+				cursor: 24,
+				aborted: false,
+				truncated: false
+			}
+		}
+	},
+	{
+		description: "Using reserved regex characters as quote characters",
+		input: '.a\na.,b\r\nc,d\r\ne,f\r\ng,h\r\ni,j',
+		config: { quoteChar: '.' },
+		expected: {
+			data: [['a\na', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h'], ['i', 'j']],
+			errors: [],
+			meta: {
+				linebreak: '\r\n',
+				delimiter: ',',
+				cursor: 27,
+				aborted: false,
+				truncated: false
+			}
+		}
 	}
 ];
 
@@ -1215,6 +1345,10 @@ describe('Parse Tests', function() {
 	function generateTest(test) {
 		(test.disabled ? it.skip : it)(test.description, function() {
 			var actual = Papa.parse(test.input, test.config);
+			// allows for testing the meta object if present in the test
+			if (test.expected.meta) {
+				assert.deepEqual(actual.meta, test.expected.meta);
+			}
 			assert.deepEqual(JSON.stringify(actual.errors), JSON.stringify(test.expected.errors));
 			assert.deepEqual(actual.data, test.expected.data);
 		});