[Bug] Dashboard takes up a lot of Memory
### Search before asking

- [X] I searched the [issues](https://github.com/ray-project/ray/issues) and found no similar issues.


### Ray Component

Dashboard

### Issue Severity

Medium: It contributes to significant difficulty to complete my task but I work arounds and get it resolved.

### What happened + What you expected to happen

We are running Ray cluster on Kubernetes.
Ray dashboard takes up 21Gb memory causing memory overload on Ray head.

![image](https://user-images.githubusercontent.com/54840261/161507028-32e264ea-401a-4313-aa5c-88b9cc443f2a.png)
![image](https://user-images.githubusercontent.com/54840261/161507052-717bb464-2651-4867-8ba5-eb3a075867d1.png)

```
root&#64;ray-ray-head-type-qgg87:/data# pmap -x 3240459
3240459:   /usr/local/bin/python -u /usr/local/lib/python3.7/site-packages/ray/dashboard/dashboard.py --host=0.0.0.0 --port=8265 --port-retries=0 --redis-address=192.168.4.7:6379 --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2022-03-08_08-31-04_272087_3240181/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password 5241590000000000
Address           Kbytes     RSS   Dirty Mode  Mapping
000055ed1bf63000       4       4       0 r---- python3.7
000055ed1bf64000       4       4       0 r-x-- python3.7
000055ed1bf65000       4       0       0 r---- python3.7
000055ed1bf66000       4       4       4 r---- python3.7
000055ed1bf67000       4       4       4 rw--- python3.7
000055ed1dc9f000 16949244 16949028 16949028 rw---   [ anon ]
00007fbbde5a9000     256     256     256 rw---   [ anon ]
00007fbbde6a9000     256     256     256 rw---   [ anon ]
00007fbbdf237000   43532   43532   43532 rw---   [ anon ]
00007fbbe1cde000     768     768     768 rw---   [ anon ]
00007fbbe1dde000   18944   18944   18944 rw---   [ anon ]
00007fbbe3080000   81408   81408   81408 rw---   [ anon ]
00007fbbe8000000     132      16      16 rw---   [ anon ]
00007fbbe8021000   65404       0       0 -----   [ anon ]
00007fbbec002000   15872   15872   15872 rw---   [ anon ]
00007fbbecf9e000   14848   14848   14848 rw---   [ anon ]
00007fbbede2a000   33024   33024   33024 rw---   [ anon ]
00007fbbefe80000    1536    1536    1536 rw---   [ anon ]
00007fbbf0000000     132      16      16 rw---   [ anon ]
00007fbbf0021000   65404       0       0 -----   [ anon ]
00007fbbf4000000     132      28      28 rw---   [ anon ]
00007fbbf4021000   65404       0       0 -----   [ anon ]
00007fbbf8000000     132      20      20 rw---   [ anon ]
00007fbbf8021000   65404       0       0 -----   [ anon ]
00007fbbfc000000     132      16      16 rw---   [ anon ]
00007fbbfc021000   65404       0       0 -----   [ anon ]
00007fbc00000000     132      20      20 rw---   [ anon ]
00007fbc00021000   65404       0       0 -----   [ anon ]
00007fbc04021000     768     768     768 rw---   [ anon ]
00007fbc040f2000    9984    9984    9984 rw---   [ anon ]
00007fbc04aba000    4864    4864    4864 rw---   [ anon ]
00007fbc04f7a000     512     512     512 rw---   [ anon ]
00007fbc04ffa000       4       0       0 -----   [ anon ]
00007fbc04ffb000    8192      16      16 rw---   [ anon ]
00007fbc057fb000       4       0       0 -----   [ anon ]
00007fbc057fc000    8192      16      16 rw---   [ anon ]
00007fbc05ffc000       4       0       0 -----   [ anon ]
00007fbc05ffd000    8192      16      16 rw---   [ anon ]
00007fbc067fd000       4       0       0 -----   [ anon ]
00007fbc067fe000    8192      16      16 rw---   [ anon ]
00007fbc06ffe000       4       0       0 -----   [ anon ]
00007fbc06fff000    8192      16      16 rw---   [ anon ]
00007fbc077ff000       4       0       0 -----   [ anon ]
00007fbc07800000    8192      16      16 rw---   [ anon ]
00007fbc08000000     132      16      16 rw---   [ anon ]
00007fbc08021000   65404       0       0 -----   [ anon ]
00007fbc0c000000     132      20      20 rw---   [ anon ]
00007fbc0c021000   65404       0       0 -----   [ anon ]
00007fbc10000000     132      16      16 rw---   [ anon ]
00007fbc10021000   65404       0       0 -----   [ anon ]
00007fbc14000000     132      24      24 rw---   [ anon ]
00007fbc14021000   65404       0       0 -----   [ anon ]
00007fbc18000000     132      16      16 rw---   [ anon ]
00007fbc18021000   65404       0       0 -----   [ anon ]
00007fbc1c000000     132      20      20 rw---   [ anon ]
00007fbc1c021000   65404       0       0 -----   [ anon ]
00007fbc20000000     132      20      20 rw---   [ anon ]
00007fbc20021000   65404       0       0 -----   [ anon ]
00007fbc24036000     768     768     768 rw---   [ anon ]
00007fbc240f9000    7168    7168    7168 rw---   [ anon ]
00007fbc247f9000       4       0       0 -----   [ anon ]
00007fbc247fa000    8192      16      16 rw---   [ anon ]
00007fbc24ffa000       4       0       0 -----   [ anon ]
00007fbc24ffb000    8192      16      16 rw---   [ anon ]
00007fbc257fb000       4       0       0 -----   [ anon ]
00007fbc257fc000    8192      16      16 rw---   [ anon ]
00007fbc25ffc000       4       0       0 -----   [ anon ]
00007fbc25ffd000    8192      16      16 rw---   [ anon ]
00007fbc267fd000       4       0       0 -----   [ anon ]
00007fbc267fe000    8192      16      16 rw---   [ anon ]
00007fbc26ffe000       4       0       0 -----   [ anon ]
00007fbc26fff000    8192      16      16 rw---   [ anon ]
00007fbc277ff000       4       0       0 -----   [ anon ]
00007fbc27800000    8192      16      16 rw---   [ anon ]
00007fbc28000000     132      20      20 rw---   [ anon ]
00007fbc28021000   65404       0       0 -----   [ anon ]
00007fbc2c000000     132      24      24 rw---   [ anon ]
00007fbc2c021000   65404       0       0 -----   [ anon ]
00007fbc30000000     132      16      16 rw---   [ anon ]
00007fbc30021000   65404       0       0 -----   [ anon ]
00007fbc34000000     132      20      20 rw---   [ anon ]
00007fbc34021000   65404       0       0 -----   [ anon ]
00007fbc38000000     132      16      16 rw---   [ anon ]
00007fbc38021000   65404       0       0 -----   [ anon ]
00007fbc3c000000     132      16      16 rw---   [ anon ]
00007fbc3c021000   65404       0       0 -----   [ anon ]
00007fbc40000000     132      16      16 rw---   [ anon ]
00007fbc40021000   65404       0       0 -----   [ anon ]
00007fbc44039000    7936    7936    7936 rw---   [ anon ]
00007fbc447f9000       4       0       0 -----   [ anon ]
00007fbc447fa000    8192      16      16 rw---   [ anon ]
00007fbc44ffa000       4       0       0 -----   [ anon ]
00007fbc44ffb000    8192      16      16 rw---   [ anon ]
00007fbc457fb000       4       0       0 -----   [ anon ]
00007fbc457fc000    8192      16      16 rw---   [ anon ]
00007fbc45ffc000       4       0       0 -----   [ anon ]
00007fbc45ffd000    8192      16      16 rw---   [ anon ]
00007fbc467fd000       4       0       0 -----   [ anon ]
00007fbc467fe000    8192      16      16 rw---   [ anon ]
00007fbc46ffe000       4       0       0 -----   [ anon ]
00007fbc46fff000    8192      16      16 rw---   [ anon ]
00007fbc477ff000       4       0       0 -----   [ anon ]
00007fbc47800000    8192      16      16 rw---   [ anon ]
00007fbc48000000     132      28      28 rw---   [ anon ]
00007fbc48021000   65404       0       0 -----   [ anon ]
00007fbc4c000000     132      16      16 rw---   [ anon ]
00007fbc4c021000   65404       0       0 -----   [ anon ]
00007fbc50000000     132      16      16 rw---   [ anon ]
00007fbc50021000   65404       0       0 -----   [ anon ]
00007fbc54000000     132      20      20 rw---   [ anon ]
00007fbc54021000   65404       0       0 -----   [ anon ]
00007fbc58000000     132      16      16 rw---   [ anon ]
00007fbc58021000   65404       0       0 -----   [ anon ]
00007fbc5c000000     132      16      16 rw---   [ anon ]
00007fbc5c021000   65404       0       0 -----   [ anon ]
00007fbc60000000     132      16      16 rw---   [ anon ]
00007fbc60021000   65404       0       0 -----   [ anon ]
00007fbc64039000    7936    7936    7936 rw---   [ anon ]
00007fbc647f9000       4       0       0 -----   [ anon ]
00007fbc647fa000    8192      16      16 rw---   [ anon ]
00007fbc64ffa000       4       0       0 -----   [ anon ]
00007fbc64ffb000    8192      16      16 rw---   [ anon ]
00007fbc657fb000       4       0       0 -----   [ anon ]
00007fbc657fc000    8192      16      16 rw---   [ anon ]
00007fbc65ffc000       4       0       0 -----   [ anon ]
00007fbc65ffd000    8192      16      16 rw---   [ anon ]
00007fbc667fd000       4       0       0 -----   [ anon ]
00007fbc667fe000    8192      16      16 rw---   [ anon ]
00007fbc66ffe000       4       0       0 -----   [ anon ]
00007fbc66fff000    8192      16      16 rw---   [ anon ]
00007fbc677ff000       4       0       0 -----   [ anon ]
00007fbc67800000    8192      16      16 rw---   [ anon ]
00007fbc68000000     132      16      16 rw---   [ anon ]
00007fbc68021000   65404       0       0 -----   [ anon ]
00007fbc6c000000     132      16      16 rw---   [ anon ]
00007fbc6c021000   65404       0       0 -----   [ anon ]
00007fbc70000000     132      16      16 rw---   [ anon ]
00007fbc70021000   65404       0       0 -----   [ anon ]
00007fbc74000000     132      20      20 rw---   [ anon ]
00007fbc74021000   65404       0       0 -----   [ anon ]
00007fbc78000000     132      16      16 rw---   [ anon ]
00007fbc78021000   65404       0       0 -----   [ anon ]
00007fbc7c000000     132      16      16 rw---   [ anon ]
00007fbc7c021000   65404       0       0 -----   [ anon ]
00007fbc80000000     132      20      20 rw---   [ anon ]
00007fbc80021000   65404       0       0 -----   [ anon ]
00007fbc84039000    7936    7936    7936 rw---   [ anon ]
00007fbc847f9000       4       0       0 -----   [ anon ]
00007fbc847fa000    8192      16      16 rw---   [ anon ]
00007fbc84ffa000       4       0       0 -----   [ anon ]
00007fbc84ffb000    8192      16      16 rw---   [ anon ]
00007fbc857fb000       4       0       0 -----   [ anon ]
00007fbc857fc000    8192      16      16 rw---   [ anon ]
00007fbc85ffc000       4       0       0 -----   [ anon ]
00007fbc85ffd000    8192      16      16 rw---   [ anon ]
00007fbc867fd000       4       0       0 -----   [ anon ]
00007fbc867fe000    8192      16      16 rw---   [ anon ]
00007fbc86ffe000       4       0       0 -----   [ anon ]
00007fbc86fff000    8192      16      16 rw---   [ anon ]
00007fbc877ff000       4       0       0 -----   [ anon ]
00007fbc87800000    8192      16      16 rw---   [ anon ]
00007fbc88000000     132      16      16 rw---   [ anon ]
00007fbc88021000   65404       0       0 -----   [ anon ]
00007fbc8c000000     132      16      16 rw---   [ anon ]
00007fbc8c021000   65404       0       0 -----   [ anon ]
00007fbc90000000     132      36      36 rw---   [ anon ]
00007fbc90021000   65404       0       0 -----   [ anon ]
00007fbc94000000     132      20      20 rw---   [ anon ]
00007fbc94021000   65404       0       0 -----   [ anon ]
00007fbc98000000     296     296     296 rw---   [ anon ]
00007fbc9804a000   65240       0       0 -----   [ anon ]
00007fbc9c000000     132      16      16 rw---   [ anon ]
00007fbc9c021000   65404       0       0 -----   [ anon ]
00007fbca0000000     132      16      16 rw---   [ anon ]
00007fbca0021000   65404       0       0 -----   [ anon ]
00007fbca400f000    1024    1024    1024 rw---   [ anon ]
00007fbca410f000       4       0       0 -----   [ anon ]
00007fbca4110000    8192      16      16 rw---   [ anon ]
00007fbca4910000       4       0       0 -----   [ anon ]
00007fbca4911000    8192      16      16 rw---   [ anon ]
00007fbca5111000       4       0       0 -----   [ anon ]
00007fbca5112000    8448     272     272 rw---   [ anon ]
00007fbca5952000       4       0       0 -----   [ anon ]
00007fbca5953000    8192      16      16 rw---   [ anon ]
00007fbca6153000       4       0       0 -----   [ anon ]
00007fbca6154000    8192      16      16 rw---   [ anon ]
00007fbca6954000       4       0       0 -----   [ anon ]
00007fbca6955000    8448     276     276 rw---   [ anon ]
00007fbca7195000       4       0       0 -----   [ anon ]
00007fbca7196000    9984    1808    1808 rw---   [ anon ]
00007fbca7b56000      76      76       0 r-x-- testing.cpython-37m-x86_64-linux-gnu.so
00007fbca7b69000    2044       0       0 ----- testing.cpython-37m-x86_64-linux-gnu.so
00007fbca7d68000      12      12      12 rw--- testing.cpython-37m-x86_64-linux-gnu.so
00007fbca7d6b000     512     512     512 rw---   [ anon ]
00007fbca7deb000      80      80       0 r-x-- json.cpython-37m-x86_64-linux-gnu.so
00007fbca7dff000    2048       0       0 ----- json.cpython-37m-x86_64-linux-gnu.so
00007fbca7fff000       4       4       4 rw--- json.cpython-37m-x86_64-linux-gnu.so
00007fbca8000000     132      12      12 rw---   [ anon ]
00007fbca8021000   65404       0       0 -----   [ anon ]
00007fbcac006000    1280    1280    1280 rw---   [ anon ]
00007fbcac146000     488     424       0 r-x-- parsers.cpython-37m-x86_64-linux-gnu.so
00007fbcac1c0000    2048       0       0 ----- parsers.cpython-37m-x86_64-linux-gnu.so
00007fbcac3c0000      28      28      28 rw--- parsers.cpython-37m-x86_64-linux-gnu.so
00007fbcac3c7000     780     780     780 rw---   [ anon ]
00007fbcac48a000     276     148       0 r-x-- reduction.cpython-37m-x86_64-linux-gnu.so
00007fbcac4cf000    2048       0       0 ----- reduction.cpython-37m-x86_64-linux-gnu.so
00007fbcac6cf000      20      20      20 rw--- reduction.cpython...
```

### Versions / Dependencies

Ray v1.9.1
Kubernetes: v1.21

### Reproduction script

It happens when running ray for a long time.

### Anything else

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!
[Bug] Dashboard memory leak
### Search before asking

- [X] I searched the [issues](https://github.com/ray-project/ray/issues) and found no similar issues.


### Ray Component

Dashboard

### What happened + What you expected to happen

Ray dashboard keeps the whole history of actors and never delete the old ones. So as time going, it'll OOM.

This issue is a follow up for this one https://github.com/ray-project/ray/issues/21432

many drivers test eventually OOM because we are running faster than before and as dashboard memory keeps increasing, it eventually OOM.

### Versions / Dependencies

master

### Reproduction script

many driver nightly tests

### Anything else

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!
[Dashboard] Too many logs causes API requests (e.g. `job submit`, `job stop`) to fail
### What happened + What you expected to happen

Start Ray and run the following script on the head node to produce a lot of logs:

```python
import ray

ray.init(address="auto")

@ray.remote
def f():
    print("Print from task")

while True:
    print("Print from driver")
    ray.get(f.remote())
```



This causes Dashboard API requests to fail.  For example, submitting a job with Ray Job Submission:

```
❯ ray job submit -- sleep 30s
[...]
Traceback (most recent call last):
  File "/Users/archit/anaconda3/envs/test39/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/ray/scripts/scripts.py", line 2269, in main
    return cli()
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/ray/autoscaler/_private/cli_logger.py", line 808, in wrapper
    return f(*args, **kwargs)
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/ray/dashboard/modules/job/cli.py", line 152, in submit
    job_id = client.submit_job(
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/ray/dashboard/modules/job/sdk.py", line 142, in submit_job
    self._raise_error(r)
  File "/Users/archit/anaconda3/envs/test39/lib/python3.9/site-packages/ray/dashboard/modules/dashboard_sdk.py", line 217, in _raise_error
    raise RuntimeError(
RuntimeError: Request failed with status code 504: upstream request timeout.
```


### Versions / Dependencies

Verified on Ray 1.10 Linux and Ray 1.12 Linux, Python 3.7 and 3.9

### Reproduction script

Given above.  Here is the py-spy trace indicating the problematic part of the code:

This overloads the dashboard, giving a py-spy trace like the following: 

<img width="903" alt="Screen Shot 2022-04-27 at 12 56 08 PM" src="https://user-images.githubusercontent.com/5459654/165620600-ccc08646-b888-49d9-854e-e866912aa6f3.png">

### Issue Severity

Medium: It is a significant difficulty but I can work around it.