Unable to allocate 24.0 GiB for an array ... But I have 64 GiB of memory

### Describe the bug

I have enough memory in my system, but I can fit my model

### Steps/Code to Reproduce

```
# X has 373 columns and 1.1 million rows
# Y has just 1 column and 1.1 million rows
def train(X,Y):
    from sklearn.model_selection import train_test_split

    X_train, X_test, Y_train, Y_test = train_test_split(
        X, Y, test_size=0.3, random_state=42
    )

    from sklearn.linear_model import LogisticRegression
    from sklearn.metrics import classification_report

    model = LogisticRegression()
    model.fit(X_train, Y_train)

    predictions = model.predict(X_test)

    print(classification_report(Y_test, predictions))
```

### Expected Results

It should have trained the model

### Actual Results
```
L:\ml\inference-local-main>python applatest.py
L:\ml\inference-local-main\applatest.py:76: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  X["Date"] = pd.to_datetime(X["Date"])
L:\ml\inference-local-main\applatest.py:77: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  X["Date"] = X["Date"].dt.strftime("%Y%m%d").astype(float)
Traceback (most recent call last):
  File "L:\ml\inference-local-main\applatest.py", line 136, in <module>
    coffcients = iterate()
                 ^^^^^^^^^
  File "L:\ml\inference-local-main\applatest.py", line 130, in iterate
    coffecient_output = train(X,Y)
                        ^^^^^^^^^^
  File "L:\ml\inference-local-main\applatest.py", line 111, in train
    model.fit(X_train, Y_train)
  File "C:\Python312\Lib\site-packages\sklearn\base.py", line 1474, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\sklearn\linear_model\_logistic.py", line 1201, in fit
    X, y = self._validate_data(
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\sklearn\base.py", line 650, in _validate_data
    X, y = check_X_y(X, y, **check_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\sklearn\utils\validation.py", line 1263, in check_X_y
    X = check_array(
        ^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\sklearn\utils\validation.py", line 997, in check_array
    array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\sklearn\utils\_array_api.py", line 521, in _asarray_with_order
    array = numpy.asarray(array, order=order, dtype=dtype)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 24.0 GiB for an array with shape (1101657, 2930) and data type float64
```

### Versions

```shell
System:
    python: 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
executable: C:\Python3\python.exe
   machine: Windows-11-10.0.22631-SP0

Python dependencies:
      sklearn: 1.4.2
          pip: 23.3.2
   setuptools: 68.2.2
        numpy: 1.26.1
        scipy: 1.12.0
       Cython: None
       pandas: 2.2.2
   matplotlib: 3.8.1
       joblib: 1.3.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Python3\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Python3\Lib\site-packages\numpy.libs\libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Zen

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Python3\Lib\site-packages\scipy.libs\libopenblas_v0.3.20-571-g3dec11c6-gcc_10_3_0-c2315440d6b6cef5037bad648efc8c59.dll
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Zen
```

COMMENT:
Hi @maifeeulasad,

> X_train, X_test, Y_train, Y_test = train_test_split(
        X, Y, test_size=0.3, random_state=42
    )

Here you're creating a copy of you dataset

>     model = LogisticRegression()
    model.fit(X_train, Y_train)

depending on the memory layout and dtype of your original array, LogisticRegression may create a copy as well. To not make a copy, it expects a C-ordered array of float64. In particular, if it's a dataframe, a copy will be made because the underlying array is Fortran-ordered.

In the end the total memory consumption would be 3 times 24GB > 64GB, so the memory error is kind of expected. If possible, saving the result of train_est_split and start from there would avoid 1 copy. There's unfortunately not much else we can do about it.

Labels: Performance