Repository: dask/dask

Issue:
bag.read_text linedelimiter option has unexpected behavior
Based on the documentation my assumption is dask.bag.read_text is suppose to return a bag of lines which are defined by the passed linedelimiter.  However, specifying a non-default linedelimiter has unexpected behavior.

```python
import dask.bag as db

open('test.txt','w').write('abc\nababa\n')
print(db.read_text('test.txt').take(2))
print(db.read_text('test.txt',linedelimiter='a').take(2))
print(db.read_text('test.txt',linedelimiter='a',blocksize=1).take(2))
print(db.read_text('test.txt',linedelimiter='b',blocksize=1).take(2))
```

This prints out:

```
('abc\n', 'ababa\n')
('abc\n', 'ababa\n')
('abc\n', 'a')
('ab',)
```

Keeping the default results in the expected tuple of two lines.  Changing the delimiter to 'a' produces exactly the same result, ignoring the passed linedelimiter.  Setting the blocksize to 1 shows that the passed linedelimiter may be used when creating partitions, if not items (although the first 'a' instance is not separated).  I cannot rationalize the result with the 'b' delimiter where only one item is returned from take(2).


**Environment**:

- Dask version: 2021.02.0
- Python version: 3.8.5
- Operating System: Ubuntu 20.04
- Install method (conda, pip, source): pip



Discussion:
I agree that I would expect `linedelimiter` to behave like you expected. I think there are several compounding issues here. One is that if `blocksize` is provided the funtion follows an entirely different codepath. If `blocksize` is not provided `linedelimiter` is dropped entirely.

If you are interested in fixing this behavior I think the first step would be to check that the `linedelimiter` always makes it to fsspec, and then you'd want to check to make sure that fsspec does what you'd expect with it in this file https://github.com/dask/dask/blob/master/dask/bytes/core.py
I did look at this a little and honestly I cannot figure out why the default case works the way it does (with separate lines).  fsspec.utils.read_block only uses the delimiter to make sure the block is ends on a delimiter - it doesn't split the block.  I haven't been able to find where the first example is splitting based on the newline.

I realized I had been missing a UserWarning (didn't show in Jupyter?) for the last line:
```
/home/dkoes/git/dask/dask/dask/bag/core.py:2444: UserWarning: Insufficient elements for `take`. 2 elements requested, only 1 elements available. Try passing larger `npartitions` to `take`.
```
Based on the documentation for `take`, the last line is not an error, although I would argue this design choice is unintuitive, and it begs the question why the second to last line prints two items.
`file_to_blocks` in dask.bag.text is iterating over the lines of the file.  These files are opened with fsspec.open_files without setting the newline argument to define the line delimiter to the requested values.

However, even if that is fixed, it doesn't appear that fsspec has the expected behavior (it ignores the newline argument when iterating):
```python
lazy = fsspec.open_files('test.txt',newline='a')[0]
with lazy as f:
    for line in f:
        print(line)
```
```
b'abc\n'
b'ababa\n'
```

Looking at fsspec, the newline parameter is only used if the mode is text (the default for fsspec is rb, but dask sets this to rt).  In that case it is passed to TextIOWrapper, which only supports '','\r','\n' and '\rn' (https://docs.python.org/3/library/io.html#io.TextIOWrapper).  With the default 'rb' mode, the newline argument is not passed to the file.open function call so universal newlines are used (https://docs.python.org/3/library/functions.html#open-newline-parameter).  Non-standard newlines are not supported with open either.

I don't think the dask linedelimiter can be correctly implemented unless you implement your own line buffering wrapper on top of fsspec.  Even if we only consider the standard delimiters, the linedelimiter argument is not being passed through to open_files.  So, for instance, we get:
```python
read_text('test.txt',linedelimiter='\r').take(2)
```
```
('abc\n', 'ababa\n')
```
instead of
```
('abc\nababa\n',)
```

If the call to `open_files` in `read_text` were changed to
```python
    files = open_files(
        urlpath,
        mode="rt",
        encoding=encoding,
        errors=errors,
        compression=compression,
        newline=linedelimiter,
        **(storage_options or {})
    )
```
(along with updating the documentation for the option to explicitly state what delimiters were supported and changing the name to newline to be more consistent with other python documentation) then at least informative error messages would be provided with nonstandard delimiters and it would work as expected for the standard delimiters.

This wouldn't do what I wanted it to do, which is split up a text file by an arbitrary delimiter, but that doesn't seem to be a feature that is well supported by the underlying libraries.



Note that `read_bytes` probably doesn't process the linedelimiter the exactly the same was as fsspec.open, which supports None and '' values for variants of universal newline support, so likely read_text should only support a single explicit newline: '\n','\r', and '\r\n' and error checking for these values should occur in `read_text`.
I started to put together a pull request for this, but it's proving to be a deeper rabbit hole than expected.  Here's a test (that I cannot get to pass):
```python
def test_delimiter():
    with filetexts({'.test.delim.txt': 'abc\ndef\rghi\nxyz\r\n123'}):
        assert read_text(".test.delim.txt",linedelimiter='\n').count().compute() == 4
        assert read_text(".test.delim.txt",linedelimiter='\n',blocksize=2).count().compute() == 4
        assert read_text(".test.delim.txt",linedelimiter='\r').count().compute() == 3
        assert read_text(".test.delim.txt",linedelimiter='\r',blocksize=2).count().compute() == 3
        assert read_text(".test.delim.txt",linedelimiter='\r\n').count().compute() == 2
        assert read_text(".test.delim.txt",linedelimiter='\r\n',blocksize=2).count().compute() == 3

        with pytest.raises(ValueError):
            read_text(".test.delim.txt",linedelimiter='abc')
```
There are different code paths depending if blocksize is set.  If it is set, StringIO is used to separate the data into lines.  Although StringIO takes a newline argument, it uses it both to determine lines and to _translate_ newline characters.  So it will change '\n' to '\r' and vice versa depending on the setting.  This means you will get different results with the different code paths.  I have not been able to figure out a way to split the string while keeping the delimiters intact that is as fast as using StringIO.

I think the best course of action is to either remove the linedelimiter option or rename it to blockdelimiter and carefully document it so it is clear that only effects the partition boundaries.  However the utility of such an option seems dubious - the blocks will be split differently than the lines.  Why would you want this? Lines at block boundaries could end up split into pieces.

Alternatively linedelimiter could be implemented as would be expected, with a slower code path for non `os.linesep` values.
Thank you for looking into this so carefully! On first thought, it seems to me that we should deprecate `linedelimiter` and see if there is any pushback. Since the docstring seems not to define the behavior very well and there don't seem to be any open issues about this, my guess is that no one uses it. In cases like that getting rid of odd and confusing behavior is probably the best course of action. cc @jrbourbeau who knows more about dask.bag and @martindurant who knows about fsspec :smile: 
So that we are on the same page, can we agree what the output of the original code ought to be? I think it's probably a trivial fix in `dask` and `fsspec` appears to be doing exactly what was asked of it. I am thinking to differentiate between the line delimiters that are explicitly supported by `StringIO` (`[None, '', '\n', '\r', '\r\n']`) and those that are not.

Resolving patch:
diff --git a/dask/bag/text.py b/dask/bag/text.py
--- a/dask/bag/text.py
+++ b/dask/bag/text.py
@@ -1,5 +1,4 @@
 import io
-import os
 from functools import partial
 
 from tlz import concat
@@ -18,7 +17,7 @@ def read_text(
     compression="infer",
     encoding=system_encoding,
     errors="strict",
-    linedelimiter=os.linesep,
+    linedelimiter=None,
     collection=True,
     storage_options=None,
     files_per_partition=None,
@@ -41,7 +40,7 @@ def read_text(
         Compression format like 'gzip' or 'xz'.  Defaults to 'infer'
     encoding: string
     errors: string
-    linedelimiter: string
+    linedelimiter: string or None
     collection: bool, optional
         Return dask.bag if True, or list of delayed values if false
     storage_options: dict
@@ -89,18 +88,26 @@ def read_text(
     if isinstance(blocksize, str):
         blocksize = parse_bytes(blocksize)
 
-    files = open_files(
-        urlpath,
-        mode="rt",
-        encoding=encoding,
-        errors=errors,
-        compression=compression,
-        **(storage_options or {})
-    )
     if blocksize is None:
+        if linedelimiter in [None, "", "\n", "\r", "\r\n"]:
+            newline = linedelimiter
+            linedelimiter = None
+        else:
+            newline = ""
+        files = open_files(
+            urlpath,
+            mode="rt",
+            encoding=encoding,
+            errors=errors,
+            compression=compression,
+            newline=newline,
+            **(storage_options or {})
+        )
         if files_per_partition is None:
             blocks = [
-                delayed(list)(delayed(partial(file_to_blocks, include_path))(fil))
+                delayed(partial(file_to_blocks, include_path, delimiter=linedelimiter))(
+                    fil
+                )
                 for fil in files
             ]
         else:
@@ -109,15 +116,18 @@ def read_text(
                 block_files = files[start : (start + files_per_partition)]
                 block_lines = delayed(concat)(
                     delayed(map)(
-                        partial(file_to_blocks, include_path),
+                        partial(file_to_blocks, include_path, delimiter=linedelimiter),
                         block_files,
                     )
                 )
                 blocks.append(block_lines)
     else:
+        # special case for linedelimiter=None: we will need to split on an actual bytestring
+        # and the line reader will then use "universal" mode. Just as well that \r\n and \n
+        # will both work (thankfully \r for MacOS is no longer a thing)
         o = read_bytes(
             urlpath,
-            delimiter=linedelimiter.encode(),
+            delimiter=linedelimiter.encode() if linedelimiter is not None else b"\n",
             blocksize=blocksize,
             sample=False,
             compression=compression,
@@ -125,7 +135,10 @@ def read_text(
             **(storage_options or {})
         )
         raw_blocks = o[1]
-        blocks = [delayed(decode)(b, encoding, errors) for b in concat(raw_blocks)]
+        blocks = [
+            delayed(decode)(b, encoding, errors, linedelimiter)
+            for b in concat(raw_blocks)
+        ]
         if include_path:
             paths = list(
                 concat([[path] * len(raw_blocks[i]) for i, path in enumerate(o[2])])
@@ -143,10 +156,21 @@ def read_text(
     return blocks
 
 
-def file_to_blocks(include_path, lazy_file):
+def file_to_blocks(include_path, lazy_file, delimiter=None):
+    # blocksize is None branch
     with lazy_file as f:
-        for line in f:
-            yield (line, lazy_file.path) if include_path else line
+        if delimiter is not None:
+            text = f.read()
+            if not text:
+                return []
+            parts = text.split(delimiter)
+            yield from (
+                (line, lazy_file.path) if include_path else line
+                for line in [line + delimiter for line in parts[:-1]] + parts[-1:]
+            )
+        else:
+            for line in f:
+                yield (line, lazy_file.path) if include_path else line
 
 
 def attach_path(block, path):
@@ -154,7 +178,17 @@ def attach_path(block, path):
         yield (p, path)
 
 
-def decode(block, encoding, errors):
+def decode(block, encoding, errors, line_delimiter):
+    # blocksize is not None branch
     text = block.decode(encoding, errors)
-    lines = io.StringIO(text)
-    return list(lines)
+    if line_delimiter in [None, "", "\n", "\r", "\r\n"]:
+        lines = io.StringIO(text, newline=line_delimiter)
+        return list(lines)
+    else:
+        if not text:
+            return []
+        parts = text.split(line_delimiter)
+        out = [t + line_delimiter for t in parts[:-1]] + (
+            parts[-1:] if not text.endswith(line_delimiter) else []
+        )
+        return out


Test patch:
diff --git a/dask/bag/tests/test_text.py b/dask/bag/tests/test_text.py
--- a/dask/bag/tests/test_text.py
+++ b/dask/bag/tests/test_text.py
@@ -119,3 +119,21 @@ def test_errors():
         result = read_text(".test.foo", encoding="ascii", errors="ignore")
         result = result.compute(scheduler="sync")
         assert result == ["Jos\n", "Alice"]
+
+
+def test_complex_delimiter():
+    longstr = "abc\ndef\n123\n$$$$\ndog\ncat\nfish\n\n\r\n$$$$hello"
+    with filetexts({".test.delim.txt": longstr}):
+        assert read_text(".test.delim.txt", linedelimiter="$$$$").count().compute() == 3
+        assert (
+            read_text(".test.delim.txt", linedelimiter="$$$$", blocksize=2)
+            .count()
+            .compute()
+            == 3
+        )
+        vals = read_text(".test.delim.txt", linedelimiter="$$$$").compute()
+        assert vals[-1] == "hello"
+        assert vals[0].endswith("$$$$")
+        vals = read_text(".test.delim.txt", linedelimiter="$$$$", blocksize=2).compute()
+        assert vals[-1] == "hello"
+        assert vals[0].endswith("$$$$")