Repository: dask/dask

Issue:
Weird behavior with .read_csv(header=0)
So I was testing both a pandas and a dask solution for an assignment of parsing a CSV file, when I found a weird behavior with dask.read_csv().

A) Let's assume we have the next CSV file (input.csv) with 13 unique rows.

`1 city1,date,193` < This 193 value will be important going forward
`2 city2,date,14`
`(...)`
`13 city13,date,18`

Given this CSV file, we run this script

`df = dd.read_csv("input.csv", header=0, names=["city", "date", "sales"], usecols=["city", "sales"], blocksize=16 * 1024 * 1024)`
`result = df.groupby("city").sum()`
`result = result.compute().to_csv("./output.csv")`

This give us an output CSV file with 2 columns, city and sales, and 12 rows. **The CSV has not header info in line 1**, and because of the (header=0) we're "skipping" the first line and city1 is missing. It was my error to put the header=0 there, but everything is working as intended here.

But **groupby().sum()** in the second line is not doing anything since we just have 13, unique rows. So I replicated those lines 780k times, ending with a 10.140.000 rows file.

This is where things start to get messy.

Remember, city1 have a sales value = 193. Given the new input file, the output file should have a value for city1 = 150539807. This is 193 * 779.999 (We have the same rows 780k times, but we're skipping the first line, which is a city1 row)

**However**, the result for city1 in the output file is 150542123. So it's looks like if additional 12 rows for city1 are being parsed.

What if instead of 780k times, I repeat the rows half of it (390k times)? The additional rows also halts, and additional 6 rows are parsed.

Halts again, repeat the rows 195k times, and value for city1 is 37634807, which is exactly 193 * 194.999, cause first rows is skipped. So issue disappears at this point.

But if I replicate the rows 194.999/998 times, or even adds up to 195.001/002 times, first line is not skipped and we also have an additional row in total count.

All other 12 cities values are always, in all cases, what they should be.

Why is this happening? It's really weird.

If I run the same script but just with pandas, the first line is always skipped and total count for the city rows is exactly what should be.

**Environment**:

- Dask version: 2022.10.0
- Python version: 3.11.0
- Operating System: Windows 10.0.19043
- Install method (conda, pip, source): pip



Discussion:
@cryptohabitat -- Here's an attempt at a reproducer.

```
import pandas as pd
import dask.dataframe as dd
from dask.dataframe.utils import assert_eq

# create your csv file
pdf = pd.DataFrame({1: ["city1", "date", 193],
                    2: ["city2", "date", 14],
                    3: ["city3", "date", 15],
                    4: ["city4", "date", 16],
                    5: ["city5", "date", 19],
                    6: ["city6", "date", 22],
                    7: ["city7", "date", 25],
                    8: ["city8", "date", 28],
                    9: ["city9", "date", 29],
                    10: ["city10", "date", 30],
                    11: ["city11", "date", 31],
                    12: ["city12", "date", 32],
                    13: ["city13", "date", 22],
                  )
df = pdf.T
df.columns=["city", "date", "sales"]
dfs = 780_000*[df]
df = pd.concat(dfs, axis=0)
df.to_csv("input.csv", index=None)

# Then load with both dask and pandas
df = pd.read_csv("input.csv", usecols=["city", "sales"], header=0)
df = df.groupby("city").sum()

ddf = dd.read_csv("input.csv", usecols=["city", "sales"], header=0, blocksize=16 * 1024 * 1024)
ddf = ddf.groupby("city").sum().compute()
assert_eq(ddf, df)
```

This return `True` for me on `python==3.9.13` and `dask==2022.10.1`

Also tested on `python=3.11` and `dask==2022.10.1` and it passes.

Did I miss something in your reproducer?
@cryptohabitat -- that's for the report. I'm having a little trouble following exactly what's going on here. But here's my take:

1. if you use `header=None` instead of `header=0`, everything seems to work as expected
2. However, if you do use `header=0` (which is an easy mistake to make), and when you have a CSV with some large number of rows, it appears that dask is artificially adding additional rows/partitions/something

Do I have that right?

Could I ask you to re-write your example where City1 has a value of 100 or 10 so that these number are easier to track in your issue?

<details>Things working correctly with `header=None`

```python
from io import StringIO
import pandas as pd
import dask.dataframe as dd

csv = StringIO("""\
1 city1,date,193
2 city2,date,14
13 city13,date,18
""")

N = 780_000
_df = pd.read_csv(csv, header=None)
df = pd.concat([_df for _ in range(N)], ignore_index=True)
df.to_csv("temp.csv", index=False, header=False)
               

pdf = pd.read_csv("temp.csv", header=None, names=["city", "date", "sales"], usecols=["city", "sales"])

ddf = dd.read_csv("temp.csv", header=None, names=["city", "date", "sales"],
                usecols=["city", "sales"], blocksize=16 * 1024 * 1024)
res = ddf.groupby("city").sum().compute()
print(res)
assert(res.loc["1 city1", "sales"] == 193 * N)
```

</details>
**_(Sorry, I closed the issue on a missclick. Re-opened with my answers and new code to replicate)_**

I've been doing some more test around this.

First of all, according to [pandas docs (v 1.5.1)](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html) for header :


> 'Default behaviour is to infer the column names: If no names are passed the behaviour is identical to `header=0` and columns name are inferred from the first line of the file. If column names are passed explicitly then the behaviour is identical to `header=None`. Explicitly pass `header=0` to be able to replace existing names.'

**So, just to be clear, the problem kicks in when you're using `header=0` and `names=[]` to replace the header info, but no header is being provided on line 1.**

The reason why @hayesgb script is ending with `True` when comparing the 2 data frames is because your input.CSV have header info in line 1. If header info exist in line one then there's no problem.

Now, about the @phobson answer, yes, you are right. Given the example, pandas is actually parsing the file correctly. It just skip the first line, so `city1 = value * 779.999` (Because it skipped it that first time). However, dask is adding additionals 'something', because value for city1 have ~10 additional row.

Here's the code to replicate both cases, including CSV file. I'm using 10 as initial value for city1's sales for it to be more "traceable"

<details>
<summmary>Code</summary>

```python
from io import StringIO
import pandas as pd
import dask.dataframe as dd
from dask.dataframe.utils import assert_eq

# create your csv file W/O HEADERS on line 1
csv = StringIO("""\
city1,1992-09-13,10
city2,1992-09-13,14
city3,1992-09-13,98
city4,1992-09-13,13
city5,1992-09-13,45
city6,1992-09-13,64
city7,1992-09-13,72
city8,1992-09-13,12
city9,1992-09-13,31
city10,1992-09-13,27
city11,1992-09-13,46
city12,1992-09-13,51
city13,1992-09-13,80
""")

N = 780_000
_df = pd.read_csv(csv, header=None)
df = pd.concat([_df for _ in range(N)], ignore_index=True)
df.to_csv("input.csv", index=False, header=False)

#Load with pandas
df = pd.read_csv("input.csv", header=0, names=["city", "date", "sales"], usecols=["city", "sales"], chunksize=100000)
chunks = [chunk.groupby('city').sum() for chunk in df]
df = pd.concat(chunks).groupby(level=0).sum()

#Load with dask
ddf = dd.read_csv("input.csv", header=0, names=["city", "date", "sales"], usecols=["city", "sales"], blocksize=16 * 1024 * 1024)
ddf = ddf.groupby("city").sum().compute()

#Compare both df.
print(assert_eq(ddf, df))
There is definitely a bug.  I modify a bit the reproducer  and try multiple things, and something is not right:

```python
from io import StringIO
import pandas as pd
import dask.dataframe as dd
from dask.dataframe.utils import assert_eq

# create your csv file W/O HEADERS on line 1
csv = StringIO("""\
city1,1992-09-13,10
city2,1992-09-13,14
city3,1992-09-13,98
city4,1992-09-13,13
city5,1992-09-13,45
city6,1992-09-13,64
city7,1992-09-13,72
city8,1992-09-13,12
city9,1992-09-13,31
city10,1992-09-13,27
city11,1992-09-13,46
city12,1992-09-13,51
city13,1992-09-13,80
""")

N = 780_000 #>>>> we'll get back to this number 
_df = pd.read_csv(csv, header=None)
df = pd.concat([_df for _ in range(N)], ignore_index=True)
df.to_csv("input.csv", index=False, header=False)

#Load with pandas
df_orig = pd.read_csv("input.csv", header=0, names=["city", "date", "sales"], usecols=["city", "sales"])
df = df_orig.groupby("city").agg("sum")

#Load with dask no blocksize provided 
ddf_no_bs = dd.read_csv("input.csv", header=0, names=["city", "date", "sales"], usecols=["city", "sales"])
ddf1 = ddf_no_bs.groupby("city").sum().compute()

#Load with dask with blocksize
ddf_w_bs= dd.read_csv("input.csv", header=0, names=["city", "date", "sales"], usecols=["city", "sales"], blocksize=16 * 1024 * 1024)
ddf2 = ddf_w_bs.groupby("city").sum().compute()
```

This is what I found
- N= 780_000 
Only city 1 has different values, and in the 3 cases is different. But the rest of the cities all match
```python
>>> all(df.sales[1:] == df.sales_dask_no_bs[1:])
True
>>>all(df.sales[1:] == df.sales_dask_w_bs[1:])
True
```

![Screen Shot 2022-10-31 at 5 41 22 PM](https://user-images.githubusercontent.com/7526622/199115831-272fe7d8-195b-4e19-9cd7-dc777f0b3c74.png)

- For N=200_000 

The discrepancy is still in city1 but only when we use blocksize. The rest of the cities are the same. 

![Screen Shot 2022-10-31 at 5 44 40 PM](https://user-images.githubusercontent.com/7526622/199116305-2c2a681c-ba6b-4868-b2e0-d877fc1b5ad9.png)

- For N=100_000 
Everything matches. 

- header = None, has no problems. 

I'm not sure what's going on, but something is up!

Maybe @rjzamora you might have an idea of what could be going on here? 
@ncclementi 
> This is what I found
> 
> * N= 780_000
>   Only city 1 has different values, and in the 3 cases is different. But the rest of the cities all match

Yes, only city 1, cause that's the row that is having the "weird" interaction with the `header=0` "targeting" at it in line 1.

> (...) but only when we use blocksize.

Maybe there's something happening at every block/chunk when using `blocksize`.

So I just checked every block by saving them in separates output files.

**- N = 780_000**

``` python
ddf = dd.read_csv("input.csv", header=0, names=["city", "date", "sales"], usecols=["city", "sales"], blocksize=16 * 1024 * 1024)
outputs = ddf.to_csv('blocks/part_*.csv', index=False, header=False)
```

We have 11 blocks as a result. And there's the problem:

- **part_00.csv** starts with city2 (because the first row is skipped) and ends with city1. So the first block have the same number of rows for each city.
- **part_01.csv starts with city1 again** (header?) and ends also with city1 < +1 additional row.

The same behavior in part_01.csv appears up until part_10. There we have our 10 additional rows.

The thing is, for N = 780_000, delta between pandas and dask is 11 rows. So 10 of those rows are being additionally parsed, but there's 1 extra row I don't know where is coming from.

**For N = 200_000 we have just 3 blocks:**

- **part_00.csv** starts with city2 (because the first row is skipped) and ends with city6.
- **part_01.csv starts with city1 again** and then keep counting from city7, so we have an additional row for city1. Same happens with part_02.

**For N = 100_000 we just have 1 block that start with city2 and ends with city13, so there are not additional rows being parsed**

For what is worth, I guess line 1 is being added to every block/partition as header except for the first one, and then being parsed (even with `header=0`).

Curious on why I didn't have problems with N = 195.000 while I did for N = 195.001/194.999? Didn't check partitions for N = those values.
The unexpected behavior can all be explained by the `header=0` argument. Since we *think* (incorrrectly) that the 0th row is a header, we prepend a copy of that row to the byte range used for every other partition in the dataset.  Partitions 1+ then end up including the copied row as "real" data for some reason. Still trying to figure out what the correct logic is [here](https://github.com/dask/dask/blob/ca95bfcf8072facde5a13accea9c6c7ba5881b76/dask/dataframe/io/csv.py#L121). I do know that the bug is resolved if we avoid dropping the `header` option when `names` is specified.

**Update**: Yeah, it looks like we are just trying to avoid using a something like `header=1` for partitions >0, because we have the header row prepended to the byte range, and just want "default" header behavior. However, we are failing to handle the `names`/`header=0` case. Therefore, I suggest we do something like:

```python
        if not is_first:
            write_header = True
            rest_kwargs.pop("skiprows", None)
            if rest_kwargs.get("header", 0) not in (None, 0):
                rest_kwargs.pop("header", None)  # Want default behavior
```
That's what I thought when checked each partition.

I'm not too familiar with the library code, but will check it later today. In any case, if `header=0, names=[]` then the header row should be what user pass in `names`. Header row should be dropped, no?

Behavior is almost exactly what one would expect of it: Skip first row cause we are telling headers are there. But header for partitions should be what passed in `names=[]`
>I'm not too familiar with the library code, but will check it later today. In any case, if header=0, names=[] then the header row should be what user pass in names. Header row should be dropped, no?

Yes, I think that's right. The header should be dropped, or the `header=0` setting should be preserved for every partition. The code can be a bit difficult to follow in some places, but what Dask is trying to do fhere is pretty simple:

1. Figure out which row (if any) corresponds to the header in the very first partition/byte-range
2. Copy the "header" row, and prepend it to all *other* partitions/byte-ranges
3. Modify the kwargs for all other partitions (**after the first**), to treat the first row as a header. In `main`, this is done by using the default `header=` argument in the final `pd.read_csv` call (rather than the one passed in by the user to `dd.read_csv`)

This approach *usually* makes sense, but there is obviously a bug for the case that the user passes in both `header=0` and `names=[...]`. The problem is that the default behavior for `pd.read_csv` is to "infer" what the header should be, and this inference depends on the `names` setting (if `names` is specified, the first row will be treated as real data - leading to bad results!).

Resolving patch:
diff --git a/dask/dataframe/io/csv.py b/dask/dataframe/io/csv.py
--- a/dask/dataframe/io/csv.py
+++ b/dask/dataframe/io/csv.py
@@ -116,7 +116,8 @@ def __call__(self, part):
         write_header = False
         rest_kwargs = self.kwargs.copy()
         if not is_first:
-            write_header = True
+            if rest_kwargs.get("names", None) is None:
+                write_header = True
             rest_kwargs.pop("skiprows", None)
             if rest_kwargs.get("header", 0) is not None:
                 rest_kwargs.pop("header", None)


Test patch:
diff --git a/dask/dataframe/io/tests/test_csv.py b/dask/dataframe/io/tests/test_csv.py
--- a/dask/dataframe/io/tests/test_csv.py
+++ b/dask/dataframe/io/tests/test_csv.py
@@ -1,7 +1,7 @@
 import gzip
 import os
 import warnings
-from io import BytesIO
+from io import BytesIO, StringIO
 from unittest import mock
 
 import pytest
@@ -1785,3 +1785,43 @@ def test_select_with_include_path_column(tmpdir):
     ddf = dd.read_csv(temp_path + "*.csv", include_path_column=True)
 
     assert_eq(ddf.col1, pd.concat([df.col1] * 6))
+
+
+@pytest.mark.parametrize("use_names", [True, False])
+def test_names_with_header_0(tmpdir, use_names):
+    # This test sets `blocksize` so that we will
+    # get two partitions in `dd.read_csv`. We are
+    # testing that `header=0` results in the expected
+    # behavior when `names` is also specified.
+    # See: https://github.com/dask/dask/issues/9610
+
+    csv = StringIO(
+        """\
+    city1,1992-09-13,10
+    city2,1992-09-13,14
+    city3,1992-09-13,98
+    city4,1992-09-13,13
+    city5,1992-09-13,45
+    city6,1992-09-13,64
+    """
+    )
+
+    if use_names:
+        names = ["city", "date", "sales"]
+        usecols = ["city", "sales"]
+    else:
+        names = usecols = None
+
+    path = os.path.join(str(tmpdir), "input.csv")
+    pd.read_csv(csv, header=None).to_csv(path, index=False, header=False)
+    df = pd.read_csv(path, header=0, names=names, usecols=usecols)
+    ddf = dd.read_csv(
+        path,
+        header=0,
+        names=names,
+        usecols=usecols,
+        blocksize=60,
+    )
+
+    # Result should only leave out 0th row
+    assert_eq(df, ddf, check_index=False)