Fix memory growth bug in read_csv
State: closed

The edge case where we hit powers of 2 every time during allocation can be painful.

Closes #24805.

xref #23527.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv

Comments:
I don't recall off the top of my head, but given that this regression was introduced during the RC period, do we still need to do a `whatsnew` ?  The regression was technically not part of an official release.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=h1) Report
> Merging [#24837](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/7e6ad8634ef6858664d3c9b48bb1942f23ad8890?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24837/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24837   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files         166      166           
  Lines       52382    52382           
=======================================
  Hits        48394    48394           
  Misses       3988     3988
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <ø> (ø)` | :arrow_up: |
| #single | `42.91% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=footer). Last update [7e6ad86...12aaad0](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=h1) Report
> Merging [#24837](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f4458c18287288562b21adece524fc1b046e9724?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24837/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24837   +/-   ##
=======================================
  Coverage   92.39%   92.39%           
=======================================
  Files         166      166           
  Lines       52378    52378           
=======================================
  Hits        48393    48393           
  Misses       3985     3985
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <ø> (ø)` | :arrow_up: |
| #single | `42.9% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=footer). Last update [f4458c1...0c366a8](https://codecov.io/gh/pandas-dev/pandas/pull/24837?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

>  don't recall off the top of my head, but given that this regression was introduced during the RC period, do we still need to do a whatsnew ? The regression was technically not part of an official release.

no i think this is ok as is, but can you add a memory asv here.
@jreback : Added an ASV as requested, and all is still green.  PTAL.
thanks @gfyoung