Repository: dask/distributed

Issue:
client.get_worker_logs() returns logs oldest first but should be newest first
```python
from dask.distributed import LocalCluster, Client
import logging

cluster = LocalCluster(n_workers=2)
client = Client(cluster)


def write_log(n):
    logger = logging.getLogger('distributed.worker')
    logger.info(f"Running {n} on worker.")
    return n
client.map(write_log, range(10))


# prints 2 oldest logs on each worker
print(client.get_worker_logs(2))

# prints full logs on each worker, with oldest first
print(client.get_worker_logs())

client.close()
cluster.close()
```

**Describe the issue**:
Dask distributed `client.get_worker_logs()` [docstring](https://distributed.dask.org/en/stable/api.html#distributed.Client.get_worker_logs) claims that it returns a “Dictionary mapping worker address to logs. Logs are returned in reversed order (newest first)”. However, it seems to return oldest logs first, and when I limit number of logs returned using “n” parameter it gives me the oldest "n" logs instead of the newest "n" logs.

This would be much more useful if it showed newest logs first, and could limit to newest “n” logs (as I think is intended behavior). Currently, logs with hundreds of lines are difficult to parse with this function. 

**Minimal Complete Verifiable Example**:

```python
from dask.distributed import LocalCluster, Client
import logging

cluster = LocalCluster(n_workers=1)
client = Client(cluster)


def write_log(n):
    logger = logging.getLogger('distributed.worker')
    logger.info(f"Running {n} on worker.")
    return n
client.map(write_log, range(10))


# prints 2 oldest logs on each worker
print(client.get_worker_logs(2))

# prints full logs on each worker, with oldest first
print(client.get_worker_logs())

client.close()
cluster.close()
```
This prints outputs of 
```
{'tcp://127.0.0.1:46867': (('INFO',
   '2023-01-12 23:00:31,760 - distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:46867'),
  ('INFO',
   '2023-01-12 23:00:31,760 - distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:46867'))}
```
and 
```
{'tcp://127.0.0.1:33455': (('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:33455'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:33455'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO -           Worker name:                          0'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO -          dashboard at:            127.0.0.1:43247'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:33061'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO -               Threads:                          4'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO -                Memory:                  32.00 GiB'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO -       Local Directory: /tmp/dask-worker-space/worker-m8z6xlla'),
  ('INFO',
   '2023-01-12 23:00:54,643 - distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   '2023-01-12 23:00:55,011 - distributed.worker - INFO -         Registered to:      tcp://127.0.0.1:33061'),
  ('INFO',
   '2023-01-12 23:00:55,011 - distributed.worker - INFO - -------------------------------------------------'),
  ('INFO',
   '2023-01-12 23:00:55,038 - distributed.worker - INFO - Running 0 on worker.'),
  ('INFO',
   '2023-01-12 23:00:55,038 - distributed.worker - INFO - Running 1 on worker.'),
  ('INFO',
   '2023-01-12 23:00:55,038 - distributed.worker - INFO - Running 2 on worker.'),
  ('INFO',
   '2023-01-12 23:00:55,039 - distributed.worker - INFO - Running 3 on worker.'),
  ('INFO',
   '2023-01-12 23:00:55,045 - distributed.worker - INFO - Running 4 on worker.'),
  ('INFO',
   '2023-01-12 23:00:55,045 - distributed.worker - INFO - Running 5 on worker.'),
  ('INFO',
   '2023-01-12 23:00:55,045 - distributed.worker - INFO - Running 6 on worker.'),
  ('INFO',
   '2023-01-12 23:00:55,045 - distributed.worker - INFO - Running 7 on worker.'),
  ('INFO',
   '2023-01-12 23:00:55,046 - distributed.worker - INFO - Running 8 on worker.'))}
```
**Anything else we need to know?**:

**Environment**:

- Dask version: 2022.12.1 ( also verified same behavior on 2022.11.1 )
- Python version: 3.9.15
- Operating System: jupyterhub running on 'Linux-5.4.219-126.411.amzn2.x86_64-x86_64-with-glibc2.31'
- Install method (conda, pip, source): pip



Discussion:
Thanks for the issue @nknezek, I'm able to reproduce. The first thing I would try would be to add a `reversed()` call here where we loop over the log `deque`

https://github.com/dask/distributed/blob/6015a2c40abeea631895dba69859c150503e566e/distributed/node.py#L117

Resolving patch:
diff --git a/distributed/node.py b/distributed/node.py
--- a/distributed/node.py
+++ b/distributed/node.py
@@ -109,12 +109,12 @@ def get_logs(self, start=0, n=None, timestamps=False):
 
         Returns
         -------
-        List of tuples containing the log level, message, and (optional) timestamp for each filtered entry
+        List of tuples containing the log level, message, and (optional) timestamp for each filtered entry, newest first
         """
         deque_handler = self._deque_handler
 
         L = []
-        for count, msg in enumerate(deque_handler.deque):
+        for count, msg in enumerate(reversed(deque_handler.deque)):
             if n and count >= n or msg.created < start:
                 break
             if timestamps:


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -5749,10 +5749,10 @@ def on_worker(dask_worker):
     await c.run(on_worker)
     logs = await c.get_worker_logs()
     logs = [row[1].partition(" - ")[2] for row in logs[a.worker_address]]
-    assert logs[-3:] == [
-        "distributed.worker - INFO - AAA",
-        "distributed.worker.state_machine - INFO - BBB",
+    assert logs[:3] == [
         "distributed.worker.memory - INFO - CCC",
+        "distributed.worker.state_machine - INFO - BBB",
+        "distributed.worker - INFO - AAA",
     ]
 
     def on_nanny(dask_worker):
@@ -5764,9 +5764,9 @@ def on_nanny(dask_worker):
     await c.run(on_nanny, nanny=True)
     logs = await c.get_worker_logs(nanny=True)
     logs = [row[1].partition(" - ")[2] for row in logs[a.worker_address]]
-    assert logs[-2:] == [
-        "distributed.nanny - INFO - DDD",
+    assert logs[:2] == [
         "distributed.nanny.memory - INFO - EEE",
+        "distributed.nanny - INFO - DDD",
     ]