Repository: wandb/wandb

Issue:
[Q] wandb stream ID error
I tried running using wandb with optuna. Most of the times it works but some times the following error occurs and this code runs but the results are not sent to the wandb website (only the system information like CPU usage and so on gets sent)

```
Exception in thread SockSrvRdThr:...
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/site-packages/wandb/sdk/service/server_sock.py", line 112, in run
    shandler(sreq)
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/site-packages/wandb/sdk/service/server_sock.py", line 174, in server_record_publish
    iface = self._mux.get_stream(stream_id).interface
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/site-packages/wandb/sdk/service/streams.py", line 206, in get_stream
    stream = self._streams[stream_id]
KeyError: 'zyeausrv'
```
given that this is  caused by `wandb` I was wondering if this was a wandb issue! 

(I tried as https://github.com/wandb/wandb/issues/3223#issuecomment-1032820724 says, by setting `os.environ["WANDB_START_METHOD"]="thread"` but the same problem occurred)


Discussion:
Running into the same issue. 

I have tried the following but the issue persists. 

> (I tried as https://github.com/wandb/wandb/issues/3223#issuecomment-1032820724 says, by setting os.environ["WANDB_START_METHOD"]="thread" but the same problem occurred)

Here's the setup I am using. 

1. First, clone the Diffusers repo: `git clone https://github.com/huggingface/diffusers`.
2. Then head to examples folder: `cd examples/text_to_image`. 
3. Install the dependencies. 
4. Now, run:

   ```bash
	export MODEL_NAME="CompVis/stable-diffusion-v1-4"
	export DATASET_NAME="lambdalabs/pokemon-blip-captions"


	accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
	  --pretrained_model_name_or_path=$MODEL_NAME \
	  --dataset_name=$DATASET_NAME --caption_column="text" \
	  --resolution=512 --random_flip \
	  --train_batch_size=1 \
	  --max_train_steps=5 \
	  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
	  --seed=42 \
	  --validation_prompt="cute creature" \
	  --report_to="wandb" \
	  --output_dir="sd-pokemon-model-lora" && sudo shutdown now
  ```

A GPU like V100 should be sufficient to reproduce the bug. 
Also having the same error here!
WandB Internal User commented: 
emilysilcock commented: 
Also having the same error here!

I think the same issue here...

```
Exception in thread SockSrvRdThr:
Traceback (most recent call last):
  File "C:\Users\absde\miniconda3\envs\vit_v1_env\lib\threading.py", line 1016, in _bootstrap_inner
Sanity Checking: 0it [00:00, ?it/s]    self.run()
  File "C:\Users\absde\miniconda3\envs\vit_v1_env\lib\site-packages\wandb\sdk\service\server_sock.py", line 112, in run
    shandler(sreq)
  File "C:\Users\absde\miniconda3\envs\vit_v1_env\lib\site-packages\wandb\sdk\service\server_sock.py", line 174, in server_record_publish
    iface = self._mux.get_stream(stream_id).interface
  File "C:\Users\absde\miniconda3\envs\vit_v1_env\lib\site-packages\wandb\sdk\service\streams.py", line 206, in get_stream
    stream = self._streams[stream_id]
KeyError: 'd5mbguuf'
```
Hi guys! Could you send debug logs and debug internal logs for the runs that are crashing?
I got the same as what was posted in the first comment. 
@ArtsiomWB I am also getting the same as in the first comment - the other thing to add is that the runs are actually crashing, just stopping, so I don't get a debug log or crash report. 

It seems to happen quite randomly - I'm running a sweep and it's stopping, for example, on the 4th run of the sweep, 21% of the way through the run - and running fine until there. I've tried running it multiple times and it always stops, but it doesn't seem to be at any predictable run or point. 

I actually left it running and it's still going (after a week or so) but hasn't done anything. There's a bit of an addition to the terminal output, but I don't think it's super helpful, but pasting below in case it is. 

When looking for the logs, there is no directory at ./wandb/run-20230202_094826-f6n0h5to (although I have the logs for plenty of other runs at the same location), which tied up with the "key error" error message. It seems that somehow the run log is being deleted mid-process? I have the relevant config file created in the sweep directory though. 

Thanks!


```
 21%|██████████████████████████████▊                                                         | 3400/16089 [07:39<28:35,  7.40it/s]
Exception in thread SockSrvRdThr:                                                                                                                                                           
Traceback (most recent call last):                                                                                                                                                            
    File "/home/silcock/.conda/envs/ner/lib/python3.8/threading.py", line 932, in _bootstrap_inner                                                                                                
        self.run()                                                                                                                                                                                
    File "/home/silcock/.conda/envs/ner/lib/python3.8/site-packages/wandb/sdk/service/server_sock.py", line 112, in run                                                                           
        shandler(sreq)                                                                                                                                                                            
    File "/home/silcock/.conda/envs/ner/lib/python3.8/site-packages/wandb/sdk/service/server_sock.py", line 174, in server_record_publish                                                         
        iface = self._mux.get_stream(stream_id).interface                                                                                                                                         
    File "/home/silcock/.conda/envs/ner/lib/python3.8/site-packages/wandb/sdk/service/streams.py", line 206, in get_stream                                                                        
        stream = self._streams[stream_id]                                                                                                                                                       
KeyError: 'f6n0h5to'                                                                                                                                                                        502 
response executing GraphQL.                                                                                                                                                                                                                                                                                                                                                         

<html><head>                                                                                                                                                                                <meta http-equiv="content-type" content="text/html;charset=utf-8">                                                                                                                          <title>502 Server Error</title>                                                                                                                                                             </head>                                                                                                                                                                                     <body text=#000000 bgcolor=#ffffff><h1>Error: Server Error</h1><h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2><h2></h2></body></html>

502 response executing GraphQL.

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">                                                                                                                          <title>502 Server Error</title></head><body text=#000000 bgcolor=#ffffff><h1>Error: Server Error</h1><h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2><h2></h2></body></html>

500 response executing GraphQL.
{"errors":[{"message":"context deadline exceeded"}]}
wandb: ERROR Error while calling W&B API: context deadline exceeded (<Response [500]>)
500 response executing GraphQL.
{"error":"internal database error"}
500 response executing GraphQL.
{"errors":[{"message":"internal database error","path":["agentHeartbeat"]}],"data":{"agentHeartbeat":null}}
wandb: ERROR Error while calling W&B API: internal database error (<Response [500]>)
500 response executing GraphQL.
{"error":"internal database error"}
wandb: Network error (HTTPError), entering retry loop.
500 response executing GraphQL.
{"errors":[{"message":"internal database error","path":["agentHeartbeat"]}],"data":{"agentHeartbeat":null}}
wandb: ERROR Error while calling W&B API: internal database error (<Response [500]>)
500 response executing GraphQL.
{"error":"internal database error"}
500 response executing GraphQL.
{"error":"internal database error"}
500 response executing GraphQL.
```

It continues like this a bit and then I get some time out errors. 
Strange that you don't have the debug logs. Can @dyhan316 send them to me? 

@emilysilcock do you happen to have logs from any other runs that have crashed? Or does it always remove them for you?
> Running into the same issue.
> 
> I have tried the following but the issue persists.
> 
> > (I tried as [#3223 (comment)](https://github.com/wandb/wandb/issues/3223#issuecomment-1032820724) says, by setting os.environ["WANDB_START_METHOD"]="thread" but the same problem occurred)
> 
> Here's the setup I am using.
> 
> 1. First, clone the Diffusers repo: `git clone https://github.com/huggingface/diffusers`.
> 2. Then head to examples folder: `cd examples/text_to_image`.
> 3. Install the dependencies.
> 4. Now, run:
>    ```shell
>     export MODEL_NAME="CompVis/stable-diffusion-v1-4"
>     export DATASET_NAME="lambdalabs/pokemon-blip-captions"
>    
>    
>     accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
>       --pretrained_model_name_or_path=$MODEL_NAME \
>       --dataset_name=$DATASET_NAME --caption_column="text" \
>       --resolution=512 --random_flip \
>       --train_batch_size=1 \
>       --max_train_steps=5 \
>       --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
>       --seed=42 \
>       --validation_prompt="cute creature" \
>       --report_to="wandb" \
>       --output_dir="sd-pokemon-model-lora" && sudo shutdown now
>    ```
> 
> ```
> 
> A GPU like V100 should be sufficient to reproduce the bug. 
> ```

I got similar error running dreambooth on v100:

```
wandb: Waiting for W&B process to finish... (success).
wandb: 
wandb: Run history:
wandb: loss ▁▁▂▃▁▂▂█▁▃▂▃▂▂▁▁▁▂▂▁▁▃▂▁▂▁▁▁▂▂▁▁▁▁▁▂▂▁▁▁
wandb:   lr ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: 
wandb: Run summary:
wandb: loss 0.02268
wandb:   lr 1e-05
wandb: 
wandb: 🚀 View run enchanting-darling-2 at: https://wandb.ai/harrywang/dreambooth/runs/apibzg5w
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20230214_214619-apibzg5w/logs
Steps: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 400/400 [04:05<00:00,  1.63it/s, loss=0.0227, lr=5e-6]
Exception in thread SockSrvRdThr:
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.10.9/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/data/User/wangjiannan/lora-sd-finetune/venv/lib/python3.10/site-packages/wandb/sdk/service/server_sock.py", line 112, in run
    shandler(sreq)
  File "/data/User/wangjiannan/lora-sd-finetune/venv/lib/python3.10/site-packages/wandb/sdk/service/server_sock.py", line 174, in server_record_publish
    iface = self._mux.get_stream(stream_id).interface
  File "/data/User/wangjiannan/lora-sd-finetune/venv/lib/python3.10/site-packages/wandb/sdk/service/streams.py", line 206, in get_stream
    stream = self._streams[stream_id]
KeyError: 'apibzg5w'
```
@ArtsiomWB see log files.
[debug.log](https://github.com/wandb/wandb/files/10738816/debug.log)
[debug-internal.log](https://github.com/wandb/wandb/files/10738817/debug-internal.log)

Artsiom Skarakhod commented: 
Hi guys!
Could you see if settins this env variable will solve this concern?

`WANDB_DISABLE_SERVICE=true`

> Artsiom Skarakhod commented: Hi guys! Could you see if settins this env variable will solve this concern?
> 
> `WANDB_DISABLE_SERVICE=true`

This solved my problem - worked! Thanks!
> Artsiom Skarakhod commented: Hi guys! Could you see if settins this env variable will solve this concern?
> 
> `WANDB_DISABLE_SERVICE=true`

This worked for me too! Thanks!
Hi, I just tried this variable setting fix. WANDB_DISABLE_SERVICE=true did not help. 
Are there any other options or workarounds?
Hi @f2010126! Send over your debug logs :)
The logs are more or less similar in the threads that reported this error. I wonder what we might be missing out here. 
Hi @dyhan316 , @sayakpaul , @emilysilcock, @f2010126, @harrywang 

could you please share which version of wandb you are using? We had one issue that manifested in the above way that we solved in the latest release (0.13.10) and i wanted to verify if this is the same issue. Also if you could provide a reproduction that would further help us debug this issue. 

I'm still looking into @harrywang repro and hopefully will be able to reproduce it on my end.

Resolving patch:
diff --git a/wandb/sdk/wandb_init.py b/wandb/sdk/wandb_init.py
--- a/wandb/sdk/wandb_init.py
+++ b/wandb/sdk/wandb_init.py
@@ -866,8 +866,9 @@ def _attach(
         raise UsageError("logger is not initialized")
 
     manager = _wl._get_manager()
-    if manager:
-        response = manager._inform_attach(attach_id=attach_id)
+    response = manager._inform_attach(attach_id=attach_id) if manager else None
+    if response is None:
+        raise UsageError(f"Unable to attach to run {attach_id}")
 
     settings: Settings = copy.copy(_wl._settings)
     settings.update(
diff --git a/wandb/sdk/wandb_manager.py b/wandb/sdk/wandb_manager.py
--- a/wandb/sdk/wandb_manager.py
+++ b/wandb/sdk/wandb_manager.py
@@ -211,9 +211,12 @@ def _inform_start(self, settings: "Settings", run_id: str) -> None:
         svc_iface = self._get_service_interface()
         svc_iface._svc_inform_start(settings=settings, run_id=run_id)
 
-    def _inform_attach(self, attach_id: str) -> Dict[str, Any]:
+    def _inform_attach(self, attach_id: str) -> Optional[Dict[str, Any]]:
         svc_iface = self._get_service_interface()
-        response = svc_iface._svc_inform_attach(attach_id=attach_id)
+        try:
+            response = svc_iface._svc_inform_attach(attach_id=attach_id)
+        except Exception:
+            return None
         return settings_dict_from_pbmap(response._settings_map)
 
     def _inform_finish(self, run_id: Optional[str] = None) -> None:
diff --git a/wandb/sdk/wandb_run.py b/wandb/sdk/wandb_run.py
--- a/wandb/sdk/wandb_run.py
+++ b/wandb/sdk/wandb_run.py
@@ -11,6 +11,7 @@
 import threading
 import time
 import traceback
+import warnings
 from collections.abc import Mapping
 from dataclasses import dataclass, field
 from datetime import datetime, timedelta
@@ -334,6 +335,28 @@ def wrapper(self: Type["Run"], *args: Any, **kwargs: Any) -> Any:
 
         return wrapper
 
+    @classmethod
+    def _noop_on_finish(cls, message: str = "", only_warn: bool = False) -> Callable:
+        def decorator_fn(func: Callable) -> Callable:
+            @functools.wraps(func)
+            def wrapper_fn(self: Type["Run"], *args: Any, **kwargs: Any) -> Any:
+                if not getattr(self, "_is_finished", False):
+                    return func(self, *args, **kwargs)
+
+                default_message = (
+                    f"Run ({self.id}) is finished. The call to `{func.__name__}` will be ignored. "
+                    f"Please make sure that you are using an active run."
+                )
+                resolved_message = message or default_message
+                if only_warn:
+                    warnings.warn(resolved_message, UserWarning, stacklevel=2)
+                else:
+                    raise errors.UsageError(resolved_message)
+
+            return wrapper_fn
+
+        return decorator_fn
+
     @classmethod
     def _noop(cls, func: Callable) -> Callable:
         @functools.wraps(func)
@@ -499,6 +522,7 @@ class Run:
 
     _attach_id: Optional[str]
     _is_attached: bool
+    _is_finished: bool
     _settings: Settings
 
     _launch_artifacts: Optional[Dict[str, Any]]
@@ -637,6 +661,7 @@ def _init(
         self._iface_port = None
         self._attach_id = None
         self._is_attached = False
+        self._is_finished = False
 
         self._attach_pid = os.getpid()
 
@@ -785,7 +810,11 @@ def __getstate__(self) -> Any:
         if not _attach_id:
             return
 
-        return dict(_attach_id=self._attach_id, _init_pid=self._init_pid)
+        return dict(
+            _attach_id=_attach_id,
+            _init_pid=self._init_pid,
+            _is_finished=self._is_finished,
+        )
 
     def __setstate__(self, state: Any) -> None:
         """Set run state from a custom pickle."""
@@ -847,6 +876,7 @@ def name(self) -> Optional[str]:
         return self._run_obj.display_name
 
     @name.setter
+    @_run_decorator._noop_on_finish()
     def name(self, name: str) -> None:
         with telemetry.context(run=self) as tel:
             tel.feature.set_run_name = True
@@ -870,6 +900,7 @@ def notes(self) -> Optional[str]:
         return self._run_obj.notes
 
     @notes.setter
+    @_run_decorator._noop_on_finish()
     def notes(self, notes: str) -> None:
         self._notes = notes
         if self._backend and self._backend.interface:
@@ -888,6 +919,7 @@ def tags(self) -> Optional[Tuple]:
         return None
 
     @tags.setter
+    @_run_decorator._noop_on_finish()
     def tags(self, tags: Sequence) -> None:
         with telemetry.context(run=self) as tel:
             tel.feature.set_run_tags = True
@@ -1022,6 +1054,7 @@ def project(self) -> str:
         """Name of the W&B project associated with the run."""
         return self.project_name()
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def log_code(
         self,
@@ -1242,6 +1275,7 @@ def _repr_mimebundle_(
     ) -> Dict[str, str]:
         return {"text/html": self.to_html(hidden=True)}
 
+    @_run_decorator._noop_on_finish()
     def _config_callback(
         self,
         key: Optional[Union[Tuple[str, ...], str]] = None,
@@ -1292,6 +1326,7 @@ def _config_artifact_callback(
     def _set_config_wandb(self, key: str, val: Any) -> None:
         self._config_callback(key=("_wandb", key), val=val)
 
+    @_run_decorator._noop_on_finish()
     def _summary_update_callback(self, summary_record: SummaryRecord) -> None:
         if self._backend and self._backend.interface:
             self._backend.interface.publish_summary(summary_record)
@@ -1376,6 +1411,7 @@ def _console_callback(self, name: str, data: str) -> None:
         if self._backend and self._backend.interface:
             self._backend.interface.publish_output(name, data)
 
+    @_run_decorator._noop_on_finish(only_warn=True)
     def _console_raw_callback(self, name: str, data: str) -> None:
         # logger.info("console callback: %s, %s", name, data)
 
@@ -1517,6 +1553,7 @@ def _log(
             self._step += 1
 
     @_run_decorator._noop
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def log(
         self,
@@ -1712,6 +1749,7 @@ def log(
             )
         self._log(data=data, step=step, commit=commit)
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def save(
         self,
@@ -1872,6 +1910,7 @@ def _finish(
         if manager:
             manager._inform_finish(run_id=self._run_id)
 
+        self._is_finished = True
         # end sentry session
         wandb._sentry.end_session()
 
@@ -1887,6 +1926,7 @@ def join(self, exit_code: Optional[int] = None) -> None:
         )
         self._finish(exit_code=exit_code)
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def status(
         self,
@@ -2383,6 +2423,7 @@ def _on_final(self) -> None:
             printer=self._printer,
         )
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def define_metric(
         self,
@@ -2545,6 +2586,7 @@ def _swap_artifact_name(self, artifact_name: str, use_as: Optional[str]) -> str:
     def _detach(self) -> None:
         pass
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def link_artifact(
         self,
@@ -2588,6 +2630,7 @@ def link_artifact(
                 # TODO: implement offline mode + sync
                 raise NotImplementedError
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def use_artifact(
         self,
@@ -2695,6 +2738,7 @@ def use_artifact(
             self._backend.interface.publish_use_artifact(artifact)
         return artifact
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def log_artifact(
         self,
@@ -2731,6 +2775,7 @@ def log_artifact(
             artifact_or_path, name=name, type=type, aliases=aliases
         )
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def upsert_artifact(
         self,
@@ -2784,6 +2829,7 @@ def upsert_artifact(
             finalize=False,
         )
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def finish_artifact(
         self,
@@ -2958,6 +3004,7 @@ def _prepare_artifact(
         artifact.finalize()
         return artifact, _resolve_aliases(aliases)
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def alert(
         self,
@@ -3005,6 +3052,7 @@ def __exit__(
         self._finish(exit_code)
         return exc_type is None
 
+    @_run_decorator._noop_on_finish()
     @_run_decorator._attach
     def mark_preempting(self) -> None:
         """Mark this run as preempting.


Test patch:
diff --git a/tests/functional_tests/t0_main/console/t5_tqdm_post_finish.py b/tests/functional_tests/t0_main/console/t5_tqdm_post_finish.py
new file mode 100644
--- /dev/null
+++ b/tests/functional_tests/t0_main/console/t5_tqdm_post_finish.py
@@ -0,0 +1,8 @@
+import wandb
+from tqdm.auto import tqdm
+
+run = wandb.init()
+progress_bar = tqdm(range(5))
+progress_bar.update(1)
+run.finish()
+progress_bar.update(1)
diff --git a/tests/functional_tests/t0_main/console/t5_tqdm_post_finish.yea b/tests/functional_tests/t0_main/console/t5_tqdm_post_finish.yea
new file mode 100644
--- /dev/null
+++ b/tests/functional_tests/t0_main/console/t5_tqdm_post_finish.yea
@@ -0,0 +1,6 @@
+id: 0.console.t5_tqdm_post_finish
+plugin:
+    - wandb
+assert:
+    - :wandb:runs[0][exitcode]: 0
+
diff --git a/tests/pytest_tests/unit_tests/test_wandb_run.py b/tests/pytest_tests/unit_tests/test_wandb_run.py
--- a/tests/pytest_tests/unit_tests/test_wandb_run.py
+++ b/tests/pytest_tests/unit_tests/test_wandb_run.py
@@ -6,6 +6,7 @@
 import numpy as np
 import pytest
 import wandb
+import wandb.errors
 from wandb import wandb_sdk
 
 
@@ -251,3 +252,49 @@ def test_resumed_run_resume_file_state(mock_run, tmp_path, settings, expected):
     run._on_ready()
 
     assert tmp_file.exists() == expected
+
+
+@pytest.mark.parametrize(
+    "method, args",
+    [
+        ("alert", ["test", "test"]),
+        ("define_metric", ["test"]),
+        ("log", [{"test": 2}]),
+        ("log_code", []),
+        ("mark_preempting", []),
+        ("save", []),
+        ("status", []),
+        ("link_artifact", [wandb.Artifact("test", type="dataset"), "input"]),
+        ("use_artifact", ["test"]),
+        ("log_artifact", ["test"]),
+        ("upsert_artifact", ["test"]),
+        ("finish_artifact", ["test"]),
+    ],
+)
+def test_error_when_using_methods_of_finished_run(mock_run, method, args):
+    run = mock_run(use_magic_mock=True)
+    run.finish()
+    assert run._is_finished
+    with pytest.raises(wandb.errors.UsageError):
+        getattr(run, method)(*args)
+
+
+@pytest.mark.parametrize(
+    "attribute, value",
+    [
+        ("config", ["test", 2]),
+        ("summary", ["test", 2]),
+        ("name", "test"),
+        ("notes", "test"),
+        ("tags", "test"),
+    ],
+)
+def test_error_when_using_attributes_of_finished_run(mock_run, attribute, value):
+    run = mock_run(use_magic_mock=True)
+    run.finish()
+    assert run._is_finished
+    with pytest.raises(wandb.errors.UsageError):
+        if isinstance(value, list):
+            setattr(getattr(run, attribute), *value)
+        else:
+            setattr(run, attribute, value)