Fixing memory leaks in read_csv
State: closed

This PR fixes a memory leak in parsers.pyx detected by valgrind, and also adds some further cleanup that should avoid memory leaks on exceptions,

closes #21353

- Moved the allocation of na_hashset further down, closer to where it is used. Otherwise it will not be freed if `continue` is executed,
- Delete `na_hashset` if there is an exception,
- Also clean up the allocation inside `kset_from_list` before raising an exception.

Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23072?src=pr&el=h1) Report
> Merging [#23072](https://codecov.io/gh/pandas-dev/pandas/pull/23072?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0ab8eb26817191bb88b6fa45b02443c4aba04326?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23072/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23072?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #23072   +/-   ##
=======================================
  Coverage   92.28%   92.28%           
=======================================
  Files         161      161           
  Lines       51434    51434           
=======================================
  Hits        47467    47467           
  Misses       3967     3967
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.68% <ø> (ø)` | :arrow_up: |
| #single | `42.29% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23072?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23072?src=pr&el=footer). Last update [0ab8eb2...36c1104](https://codecov.io/gh/pandas-dev/pandas/pull/23072?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can you run the asv's for csv to see if any effects & a whatsnew note
can you run the code at the very top of the issue here and show the leak has disappeared.
@kuraga, would you be able to test this change on your reproducer from #21353?

@jreback, not sure what asv's for csv are.

This patch fixes a memory leak in some proprietary code I was working on so I can't post it here, and I don't have a standalone reproducer for my issue, sorry. The reason why I mentioned #21353 was that the leak I was seeing was also coming from read_csv.
@zhezherun i understand, look at the top of the issue. there is a script at the top. pls run that with the new version.
@zhezherun do you know, will this patch address https://github.com/pandas-dev/pandas/issues/19941 as well? Would this memory leak have been exacerbated by multiple threads, or do you think that's a different issue?
I pulled down this branch and confirmed that it does not fix #19941 (but @zhezherun if you have any guesses on what may be going on there it'd be appreciated).
@zhezherun can you add a whatsnew note in bug fixes / io section, mentioning the issue number. ping on green.
@zhezherun can you update
Added a release note. Ping on green.
@gfyoung can you rebase and fix up?
lgtm. ping on green.
@jreback : Comments addressed, and all is green.  PTAL.
Thanks all.
Hi, I am facing this issue on google compute engine (Windows Server 2012 R2 Datacenter, 64 bit). How do I fix it? I have installed the latest version of Pandas.