RandomForestClassifier RAM memory increase on newer versions

### Describe the bug

I tested the performance of my program on different sklearn versions. I found that the program uses more memory when sklearn is on the latest version compared to older versions. 
```shell
1.0.1    1449MB
0.24.2   1338MB
0.23.2   1443MB
0.22.1   1314MB
0.22     1381MB
0.21.3   503MB
0.20.3   494MB
0.19.2   591MB
```
By the way, there is no effect on the evaluation of program execution time and output results.

### Steps/Code to Reproduce


[train.zip](https://github.com/scikit-learn/scikit-learn/files/9268813/train.zip)
[test.zip](https://github.com/scikit-learn/scikit-learn/files/9268826/test.zip)

```python
import numpy as np 
import pandas as pd 
df_train = pd.read_csv("train.csv")
df_test = pd.read_csv("test.csv")
df_train.isna().sum()
df_test.isna().sum()
df_train.info()
object_type = np.dtype("O")
df_train = df_train.apply(lambda x: x.fillna("NA") if x.dtype == object_type
                                       else x.fillna(-1))
df_test = df_test.apply(lambda x: x.fillna("NA") if x.dtype == object_type
                                       else x.fillna(-1))
from sklearn.preprocessing import LabelEncoder
categorical_columns = df_train.select_dtypes("object")
for col in categorical_columns:
    encoder = LabelEncoder()
    encoded = encoder.fit_transform(df_train[col])
    df_train[col] = encoded
categorical_columns = df_test.select_dtypes("object")
for col in categorical_columns:
    encoder = LabelEncoder()
    encoded = encoder.fit_transform(df_test[col])
    df_test[col] = encoded
from sklearn.model_selection import train_test_split
X = df_train.drop(["id","target"],axis=1)
y = df_train.target
X_train, X_test, y_train, y_test = train_test_split(X,y) 
import tracemalloc
tracemalloc.start()
from sklearn.ensemble import RandomForestClassifier 
rf = RandomForestClassifier(n_jobs=-1,n_estimators=100)
rf.fit(X_train,y_train)
preds = rf.predict(X_test)
preds = rf.predict(df_test.drop("id",axis=1))
import sklearn
print(sklearn.__version__)
current3, peak3 = tracemalloc.get_traced_memory()
print("Get_dummies memory usage is {",current3 /1024/1024,"}MB; Peak memory was :{",peak3 / 1024/1024,"}MB")
```

### Expected Results

Same memory usage

### Actual Results

The newer versions spend more RAM memory .


### Versions

```shell
1.0.1
0.24.2
0.23.2
0.22.1
0.22
0.21.3
0.20.3
0.19.2
```

COMMENT:
I made a simplified benchmark that uses only NumPy to benchmark the internal of `RandomForestClassifier` only:

```python
from memory_profiler import profile


@profile
def proc():
    from sklearn.datasets import make_classification
    from sklearn.ensemble import RandomForestClassifier

    X, y = make_classification(
        n_samples=100_000, n_features=100, random_state=0
    )
    forest = RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1)
    forest.fit(X, y)
    y_pred = forest.predict(X)


if __name__ == "__main__":
    proc()
```

One can run the benchmark with:

```bash
python -m memory_profiler script.py
```

I get the following results:

### `main` branch

```
Filename: script.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     4     42.6 MiB     42.6 MiB           1   @profile
     5                                         def proc():
     6    113.4 MiB     70.8 MiB           1       from sklearn.datasets import make_classification
     7    125.9 MiB     12.6 MiB           1       from sklearn.ensemble import RandomForestClassifier
     8                                         
     9    206.8 MiB     80.8 MiB           2       X, y = make_classification(
    10    125.9 MiB      0.0 MiB           1           n_samples=100_000, n_features=100, random_state=0
    11                                             )
    12    206.8 MiB      0.0 MiB           1       forest = RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1)
    13    338.2 MiB    131.4 MiB           1       forest.fit(X, y)
    14    345.1 MiB      6.9 MiB           1       y_pred = forest.predict(X)
```

### v0.19.2

```
Filename: script.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     4     41.9 MiB     41.9 MiB           1   @profile
     5                                         def proc():
     6     86.9 MiB     45.0 MiB           1       from sklearn.datasets import make_classification
     7     91.2 MiB      4.4 MiB           1       from sklearn.ensemble import RandomForestClassifier
     8                                         
     9     91.6 MiB      0.3 MiB           1       X, y = make_classification(
    10    172.4 MiB     80.8 MiB           1           n_samples=100_000, n_features=100, random_state=0
    11                                             )
    12    172.4 MiB      0.0 MiB           1       forest = RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1)
    13    315.5 MiB    143.1 MiB           1       forest.fit(X, y)
    14    323.3 MiB      7.8 MiB           1       y_pred = forest.predict(X)
```

### Conclusion

I cannot reproduce the above results. The increment could come from something else (pandas or the ~~encoder~~).
Edit: not even the encoder because it is outside of the profiling.

Labels: module:ensemble, Needs Investigation