Repository: wandb/wandb

Issue:
[CLI]: When training in offline mode, wandb tries to connect after a while a crashes the run
### Describe the bug

<!--- Description of the issue below  -->
Hi, i'm training using pytorch lightning wrapper for wandb using offline=True. After 4h of training with no problems, my code get holded indefinitly with the following error:

```
wandb: Network error (ConnectionError), entering retry loop.
```
Apparently for some reason wandb is trying to connect to the server but this cluster isn't connected, so it holds my code execution undefinitly.


### Additional Files

Here is the debug.log:

```
2022-09-02 09:14:36,212 INFO    MainThread:48922 [wandb_setup.py:_flush():76] Configure stats pid to 48922
2022-09-02 09:14:36,212 INFO    MainThread:48922 [wandb_setup.py:_flush():76] Loading settings from /linkhome/rech/genlgm01/uey53ph/.config/wandb/settings
2022-09-02 09:14:36,212 INFO    MainThread:48922 [wandb_setup.py:_flush():76] Loading settings from /gpfsdswork/projects/rech/ipk/uey53ph/geppetto/outputs/run/2022-09-02_09-14-33/iDesigner_SCAM_512_loss_decay_kl_1e-5_crop_aug/wandb/settings
2022-09-02 09:14:36,212 INFO    MainThread:48922 [wandb_setup.py:_flush():76] Loading settings from environment variables: {'_require_service': 'True', 'mode': 'dryrun'}
2022-09-02 09:14:36,212 WARNING MainThread:48922 [wandb_setup.py:_flush():76] Could not save program above cwd: /gpfsdswork/projects/rech/ipk/uey53ph/geppetto/run.py
2022-09-02 09:14:36,212 INFO    MainThread:48922 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': None, 'program': '/gpfsdswork/projects/rech/ipk/uey53ph/geppetto/run.py'}
2022-09-02 09:14:36,212 INFO    MainThread:48922 [wandb_init.py:_log_setup():461] Logging user logs to /gpfsdswork/projects/rech/ipk/uey53ph/geppetto/wandb/wandb/offline-run-20220902_091435-1bxfauyf/logs/debug.log
2022-09-02 09:14:36,212 INFO    MainThread:48922 [wandb_init.py:_log_setup():462] Logging internal logs to /gpfsdswork/projects/rech/ipk/uey53ph/geppetto/wandb/wandb/offline-run-20220902_091435-1bxfauyf/logs/debug-internal.log
2022-09-02 09:14:36,213 INFO    MainThread:48922 [wandb_init.py:init():495] calling init triggers
2022-09-02 09:14:36,213 INFO    MainThread:48922 [wandb_init.py:init():498] wandb.init called with sweep_config: {}
config: {}
2022-09-02 09:14:36,213 INFO    MainThread:48922 [wandb_init.py:init():548] starting backend
2022-09-02 09:14:36,213 INFO    MainThread:48922 [wandb_init.py:init():552] setting up manager
2022-09-02 09:14:36,215 INFO    MainThread:48922 [backend.py:_multiprocessing_setup():97] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2022-09-02 09:14:36,219 INFO    MainThread:48922 [wandb_init.py:init():558] backend started and connected
2022-09-02 09:14:36,221 INFO    MainThread:48922 [wandb_init.py:init():636] updated telemetry
2022-09-02 09:14:36,221 INFO    MainThread:48922 [wandb_init.py:init():700] starting run threads in backend
2022-09-02 09:14:41,226 INFO    MainThread:48922 [wandb_run.py:_console_start():1962] atexit reg
2022-09-02 09:14:41,226 INFO    MainThread:48922 [wandb_run.py:_redirect():1820] redirect: SettingsConsole.WRAP_RAW
2022-09-02 09:14:41,226 INFO    MainThread:48922 [wandb_run.py:_redirect():1885] Wrapping output streams.
2022-09-02 09:14:41,226 INFO    MainThread:48922 [wandb_run.py:_redirect():1907] Redirects installed.
2022-09-02 09:14:41,227 INFO    MainThread:48922 [wandb_init.py:init():732] run started, returning control to user process
2022-09-02 09:14:43,131 INFO    MainThread:48922 [wandb_watch.py:watch():47] Watching
2022-09-02 13:31:59,251 ERROR   MainThread:48922 [retry.py:__call__():154] Retry attempt failed:
Traceback (most recent call last):
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x1541d11c8ee0>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1541d11c8ee0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/wandb/sdk/lib/retry.py", line 113, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/wandb/sdk/internal/internal_api.py", line 204, in execute
    return self.client.execute(*args, **kwargs)  # type: ignore
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/transport/requests.py", line 38, in execute
    request = requests.post(self.url, **post_args)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/gpfswork/rech/ipk/uey53ph/miniconda3/envs/scam/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1541d11c8ee0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
```

### Environment

WandB version: 0.13.2

Versions of relevant libraries: pytorch lighting 1.7.2


### Additional Context

_No response_


Discussion:
Hi @nicolas-dufour!

That is definitely odd behavior, we should not be trying to connect to the server in case the run is marked as offline. Is there a minimal reproduction of this bug that would be possible for you to share so that I can investigate this?

Thanks,
Ramit
Hi @ramit-wandb 
I'm not sure i can come out with a reproducible example since it's such a weird bug. What i've observed is the following: The bug happens systematically at the same training step in different runs. I however don't do anything special at this step. I also didn't have this problems on 0.12.4, i had just updated to 0.13.2.
Hi @nicolas-dufour !

Got it, alongside the debug.log, there should be a debug-internal.log associated to this run as well. That should have some more information about what was happening within your run. One more thing - are you running a sweep in offline mode? We don't support sweeps in offline mode since we need to communicate to the central W&B server to recommend hyperparameters.

Thanks,
Ramit
​Hi @nicolas-dufour,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Weights & Biases
Hi @ramit-wandb . I don't have the runs anymore so i cannot give you this files. I'm not doing a sweep just running a simple experiment.


I myself reverted to 0.12.4 since i don't have problems with that version . I hope this get fixed in the feature though so i can update again to more modern version.
WandB Internal User commented: 
nicolas-dufour commented: 
Hi @ramit-wandb . I don't have the runs anymore so i cannot give you this files. I'm not doing a sweep just running a simple experiment.


I myself reverted to 0.12.4 since i don't have problems with that version . I hope this get fixed in the feature though so i can update again to more modern version.


Resolving patch:
diff --git a/wandb/sdk/data_types/image.py b/wandb/sdk/data_types/image.py
--- a/wandb/sdk/data_types/image.py
+++ b/wandb/sdk/data_types/image.py
@@ -34,13 +34,16 @@
 
 
 def _server_accepts_image_filenames() -> bool:
-    from pkg_resources import parse_version
+    if util._is_offline():
+        return True
 
     # Newer versions of wandb accept large image filenames arrays
     # but older versions would have issues with this.
     max_cli_version = util._get_max_cli_version()
     if max_cli_version is None:
         return False
+    from pkg_resources import parse_version
+
     accepts_image_filenames: bool = parse_version("0.12.10") <= parse_version(
         max_cli_version
     )
@@ -525,7 +528,7 @@ def size_equals_image(image: "Image") -> bool:
             ]
         else:
             wandb.termwarn(
-                "Unable to log image array filenames. In some cases, this can prevent images from being"
+                "Unable to log image array filenames. In some cases, this can prevent images from being "
                 "viewed in the UI. Please upgrade your wandb server",
                 repeat=False,
             )


Test patch: