Repository: wandb/wandb

Issue:
[Q] WandB TensorBoard pushing data to server. (No tensorboard tab on the server)
I tried using tensorboardX, but it has a lot to change with the name of the chart.

So I'm using torch.utils.tensorboard instead.

But in the server of W&B, there is no tab for tensorboard. What am I doing wrong? 
For the record, I can see charts on the local tensorboard.

TensorBoard part:
```
from torch.utils.tensorboard import SummaryWriter
...
    # Create a run
    tb_logger = SummaryWriter(log_dir=Path(log_dir, name))
...
     # Logging
    tb_logger.add_scalar('Loss/Step Loss', loss.item(), global_step)
    tb_logger.add_scalar('Loss/Train Loss (Step)', epoch_loss_step, global_step)
    tb_logger.add_scalar('Process/Step', global_step, global_step)
    tb_logger.add_scalar('Process/Epoch', epoch, global_step)
    tb_logger.flush()
```

WandB part:
```
wandb_logger = wandb.init(
        project = "...", 
        entity = "....", 
        sync_tensorboard = True,
        name = run_name, 
        tags = [
            f'{model}', 
            f'{run_name}', 
            f'{device}'
        ],)
wandb_logger.config.update(config)
...
    # Logging
    wandb_logger.log({
        'Loss/Step_Loss':            loss.item(),
        'Process/Step':                 global_step,
        'Loss/Train_Loss_(Step)':    epoch_loss_step,
        'Process/Epoch':                epoch
    })
```


Discussion:
The tensorboard tab only works when using our cloud server.  Are you running wandb/local?
The setting is set for online and every run will sync to W&B servers if is that what you need, But I run it on my local computer.
You're not running `wandb local` anywhere right?

If your syncing to the cloud please share a link to a run that doesn't have tensorboard tab.
No for sure, I think it requires Docker which I don't have.

Of course, This is a [Link ](https://wandb.ai/rsna-bone-age/Bone-Age-RSNA/runs/ux7puz44) to a recent run, but I'm not sure you be able to see it or not.

I used Pytorch. And WandB and tensorboard together as I explain in the first place.
@vanpelt Any update?
WandB Internal User commented: 
vanpelt commented: 
You're not running `wandb local` anywhere right?

If your syncing to the cloud please share a link to a run that doesn't have tensorboard tab.

WandB Internal User commented: 
Msameim181 commented: 
@vanpelt Any update?

Hello @Msameim181,

[Looking at this section in our docs](https://docs.wandb.ai/guides/integrations/tensorboard#pytorch-and-tensorboard), can you try seeing if you can upload the Pytorch Profiler JSON as such and see if that alleviates the issue?
`wandb.save(glob.glob(f"runs/*.pt.trace.json")[0], base_path=f"runs")`
Hi, Thanks for responding.

There is no JSON Profile in 'runs' directory. 

I used this: 
``` wandb.save(glob.glob(f"tensorboard/*.pt.trace.json")[0], base_path=f"tensorboard") ```

This is my directory for tensorboard:
![image](https://user-images.githubusercontent.com/25700939/166705621-d5e9439e-2f71-45cf-8436-ac5c33de3e31.png)

Any thought?
And When I used this:
``` wandb.save(glob.glob(f"tensorboard/*.pt.trace.json")[0], base_path=f"tensorboard") ```

I got this: 
``` 
    wandb.save(glob.glob(f"tensorboard/*.pt.trace.json")[0], base_path=f"tensorboard")
IndexError: list index out of range 
```

Just in case you want to know the result.



WandB Internal User commented: 
Msameim181 commented: 
Hi, Thanks for responding.

There is no JSON Profile in 'runs' directory. 

I used this: 
``` wandb.save(glob.glob(f"tensorboard/*.pt.trace.json")[0], base_path=f"tensorboard") ```

This is my directory for tensorboard:
![image](https://user-images.githubusercontent.com/25700939/166705621-d5e9439e-2f71-45cf-8436-ac5c33de3e31.png)

Any thought?

WandB Internal User commented: 
Msameim181 commented: 
And When I used this:
``` wandb.save(glob.glob(f"tensorboard/*.pt.trace.json")[0], base_path=f"tensorboard") ```

I got this: 
``` 
    wandb.save(glob.glob(f"tensorboard/*.pt.trace.json")[0], base_path=f"tensorboard")
IndexError: list index out of range 
```

Just in case you want to know the result.

Hello @Msameim181,

The issue may lie in the fact that `wandb` may not be able to easily infer where the Tensorboard directory is set as you set it at a different location than the default logging directory as noted by the setting of a custom location here:

```
tb_logger = SummaryWriter(log_dir=Path(log_dir, name))
```

As seen from the [documentation](https://docs.wandb.ai/guides/integrations/tensorboard#how-do-i-configure-tensorboard-when-im-using-it-with-wandb) can you trying adding a Tensorboard patch pointing to this directory like so:

```
import wandb
wandb.tensorboard.patch(root_logdir="<logging_directory>")
wandb.init()
```

I have no issues running the default [Tensorboard](https://github.com/wandb/examples/blob/master/colabs/tensorboard/TensorBoard_and_Weights_and_Biases.ipynb) example from our examples repo which properly renders the Tensorboard tab.
​Hi @Msameim181 ,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.
​
Best,
Anish
​
No, nothing changed.
I tried these:
``` 
log_dir = "./tensorboard"
run_name = "test123"
tb_logger = SummaryWriter(log_dir=Path(log_dir, run_name))
```

```wandb.tensorboard.patch(root_logdir="./tensorboard")```
and
```wandb.tensorboard.patch(root_logdir=f"./tensorboard/{run_name}")```



Hello @Msameim181,

Are you able to provide a reproducible minimal example for us to look at? I am unable to run into the issues you mentioned with our various Tensorboard syncing methods.

Thank you!
Hi @ash0ts,

What do you need? Script of using both Tensorboard and WandB together?
Yes, if possible, it would be good to see the script you are using that is not logging the Tensorboard tab properly for you as I am unable to recreate your problem with any minimal scripts we have.
Sure,
These are the codes that I'm using in my projects:
WandB logger: [link](https://github.com/Msameim181/RSNA-Bone-Age/blob/main/utils/wandb_logger.py)
TensorBoard logger: [link](https://github.com/Msameim181/RSNA-Bone-Age/blob/main/utils/tensorboard_logger.py)
WandB Internal User commented: 
Msameim181 commented: 
Hi @ash0ts,

What do you need? Script od using both Tensorboard and WandB together?

@Msameim181 , do you have the `tfevents` files stored somewhere for those runs?
If yes, then you can [sync them to W&B](https://docs.wandb.ai/guides/integrations/tensorboard#syncing-previous-tensorboard-runs) as well.
- To upload the run to W&B with `tfevents`  (get a new id in case you deleted the run from your filesystem):
`python -c "import wandb; print(wandb.util.generate_id())"`
- O/P: a new run_id (which will be passed to `wandb sync` , for ex: `wandb sync /path/to/tfevents/events.out.tfevents.xxx --id run_id`
Yes, I stored all of the runs log files, Also ```tfevents```.

So, I chose one of the previous runs, I looked for ```run_id``` from the runs page.
The run link: https://wandb.ai/rsna-bone-age/Bone-Age-RSNA/runs/1uy2gz0y
The command:
```
>> wandb sync tensorboardLocal/20220523_110557_MobileNetV3_Pre/events.out.tfevents.1653287760.DESKTOP-ISIIVT7.11940.0 --id 1uy2gz0y
Found 1 tfevent files in C:\Users\mahdi\Desktop\Github_Work\1\RSNA-Bone-Age\tensorboardLocal\20220523_110557_MobileNetV3_Pre
Syncing: https://wandb.ai/rsna-bone-age/RSNA-Bone-Age/runs/1uy2gz0y ...
```

And still nothing on the run page:
![image](https://user-images.githubusercontent.com/25700939/171607028-ba5756c2-6e5c-4d92-8e5a-ebc4e961b6e2.png)

Any idea?

WandB Internal User commented: 
Msameim181 commented: 
Yes, I stored all of the runs log files, Also ```tfevents```.

So, I chose one of the previous runs, I looked for ```run_id``` from the runs page.
The run link: https://wandb.ai/rsna-bone-age/Bone-Age-RSNA/runs/1uy2gz0y
The command:
```
>> wandb sync tensorboardLocal/20220523_110557_MobileNetV3_Pre/events.out.tfevents.1653287760.DESKTOP-ISIIVT7.11940.0 --id 1uy2gz0y
Found 1 tfevent files in C:\Users\mahdi\Desktop\Github_Work\1\RSNA-Bone-Age\tensorboardLocal\20220523_110557_MobileNetV3_Pre
Syncing: https://wandb.ai/rsna-bone-age/RSNA-Bone-Age/runs/1uy2gz0y ...
```

And still nothing on the run page:
![image](https://user-images.githubusercontent.com/25700939/171607028-ba5756c2-6e5c-4d92-8e5a-ebc4e961b6e2.png)

Any idea?

Thanks for the additional context @Msameim181.
[1] is the code you're tracking in this particular run already instrumented with Tensorboard calls? I'm suspecting it might not be the case. Reason i'm asking: The intended use of the `sync_tensorboard` flag is to easily align code instrumented with Tensorboard specific calls directly to the W&B workspace - [here](https://github.com/wandb/examples/blob/master/examples/tensorboard/tensorboard-tf2-kerasfit/train.py) is a good example from our examples repo that shows what that could look like in practice. It's mainly meant to allow those who have spent a lot of time working with Tensorboard to not have to reinstrument their scripts with all W&B API calls, and just flip the `sync_tensorboard` flag to True in their call to `wandb.init()`.
[2] Could you also try adding `writer = SummaryWriter(f"runs/{experiment_name}")` (or `./tensorboard/{run_name}` in your case a custom output dir) after your `wandb.init` call? Let me know if this helps.

Hi @anmolmann,

[1] I'm not sure I completely understand your point, But here is my answer: 
I use WandB and TensorBoard side by side. (I will provide a progress schema in [2])

> It's mainly meant to allow those who have spent a lot of time working with Tensorboard to not have to reinstrument their scripts with all W&B API calls, and just flip the sync_tensorboard flag to True in their call to wandb.init().
 
You said, And It's perfect. If you tell me because you have a total fully functional WandB and synced data on the server, The Tensorboard data will be useless and we do not sync and show them, I will accept. But it has alittle bad look.
So, I hope that I answered your first question.

---

[2] I think I'm already doing that. In this [link](https://github.com/Msameim181/RSNA-Bone-Age/blob/main/utils/tensorboard_logger.py)
This is the application process schema (flowchart):
1. Getting Args
2. Loading Dataset
3. Loading Model
4. Loading hyperparameters
5. Start Trainig Phase:
    1. Setup WandB logger
    2. Setup Tensorboard logger
    3. Epoch loop
         1. Reading data on batch size
         2. Training
         3. Validation
         4. Log each validation (WandB and Teansorboard)
         5. Log each epoch train (WandB and Teansorboard)
    4. Save checkpoints
6. Finish Training
7. Test Phase

So I already use ```SummaryWriter``` after `wandb.init`.

Let me know if you need any thing else. Thank you for your time.
Could it be that `sync_tensorboard` doesn't work on Windows? I got this [example](https://github.com/wandb/examples/blob/master/examples/tensorboard/tensorboard-tf2-gradienttape/train.py) to work on Linux (the TF tab shows up in the wandb UI) , but it doesn't work on Windows for me (the TF tab doesn't show up). I used a fresh python environment with wandb==0.12.17 and tensorflow/tensorboard==2.9.1
@loganlebanoff , it seems like you might be right. @Msameim181 , could you please confirm once if this is the case? I couldn't replicate the issue on my end. 
Also, If you could your environment? Then, I'll try again to replicate it on my end and also, file a bug report & ask help from our eng team.
@anmolmann, Well, I'm using Windows 10 in my case. I didn't try on Linux, so the Theory is half true on my side.
Environment:
Python==3.9.0
wandb==0.12.14
tensorboard==2.8.0
torch==1.11.0+cu113
WandB Internal User commented: 
loganlebanoff commented: 
Could it be that `sync_tensorboard` doesn't work on Windows? I got this [example](https://github.com/wandb/examples/blob/master/examples/tensorboard/tensorboard-tf2-gradienttape/train.py) to work on Linux (the TF tab shows up in the wandb UI) , but it doesn't work on Windows for me (the TF tab doesn't show up). I used a fresh python environment with wandb==0.12.17 and tensorflow/tensorboard==2.9.1

> @anmolmann, Well, I'm using Windows 10 in my case. I didn't try on Linux, so the Theory is half true on my side.
> Environment:
> Python==3.9.0
> wandb==0.12.14
> tensorboard==2.8.0
> torch==1.11.0+cu113

Did you get a chance to try this out on Linux platform with WSL on your Windows instead?
I think I could try. Why don't you give me a couple of days, to see what I can do? I'll give you the update shortly. Thanks.
Yes, It seems about right. On WSL works perfectly. the TF tab appears on the new run page

![image](https://user-images.githubusercontent.com/25700939/173182740-98b017a6-4004-42b3-a1c2-e8c15199ea4c.png)

Also, the TB logs seem to find a way to be in main page charts (It's a little odd but okay).

![image](https://user-images.githubusercontent.com/25700939/173182714-ec31236d-5036-4f1c-ab8e-e10bd156978a.png)

Thanks for the update @Msameim181. I'll make a note of this and file in an internal ticket for this specific issue on Windows.
@anmolmann Yeh, anytime. Hope it will be fixed.
> wandb sync tensorboardLocal/20220523_110557_MobileNetV3_Pre/events.out.tfevents.1653287760.DESKTOP-ISIIVT7.11940.0 --id 1uy2gz0y

I was trying to use this on WSL to sync tensorboard to wandb, But nothing. The status goes to sync and uploading data, but No results.

![image](https://user-images.githubusercontent.com/25700939/174546059-48932e38-8f6c-4e63-a82b-0b53e72c4547.png)

![image](https://user-images.githubusercontent.com/25700939/174546109-fccc9f5d-82ab-4cab-a7a0-5ba1e9bdd810.png)

I have the same problem where sync_tensorboard = True and I have no result in the cloud.

Resolving patch:
diff --git a/wandb/sdk/internal/tb_watcher.py b/wandb/sdk/internal/tb_watcher.py
--- a/wandb/sdk/internal/tb_watcher.py
+++ b/wandb/sdk/internal/tb_watcher.py
@@ -58,12 +58,15 @@ def _link_and_save_file(
     interface.publish_files(dict(files=[(GlobStr(glob.escape(file_name)), "live")]))
 
 
-def is_tfevents_file_created_by(path: str, hostname: str, start_time: float) -> bool:
-    """Check if a path is a tfevents file created by hostname.
+def is_tfevents_file_created_by(
+    path: str, hostname: Optional[str], start_time: Optional[float]
+) -> bool:
+    """Check if a path is a tfevents file. Optionally, check that it
+    was created by [hostname] after [start_time].
 
     tensorboard tfevents filename format:
         https://github.com/tensorflow/tensorboard/blob/f3f26b46981da5bd46a5bb93fcf02d9eb7608bc1/tensorboard/summary/writer/event_file_writer.py#L81
-    tensorflow tfevents fielname format:
+    tensorflow tfevents filename format:
         https://github.com/tensorflow/tensorflow/blob/8f597046dc30c14b5413813d02c0e0aed399c177/tensorflow/core/util/events_writer.cc#L68
     """
     if not path:
@@ -77,23 +80,27 @@ def is_tfevents_file_created_by(path: str, hostname: str, start_time: float) ->
     except ValueError:
         return False
     # check the hostname, which may have dots
-    for i, part in enumerate(hostname.split(".")):
+    if hostname is not None:
+        for i, part in enumerate(hostname.split(".")):
+            try:
+                fname_component_part = fname_components[tfevents_idx + 2 + i]
+            except IndexError:
+                return False
+            if part != fname_component_part:
+                return False
+    if start_time is not None:
         try:
-            fname_component_part = fname_components[tfevents_idx + 2 + i]
-        except IndexError:
+            created_time = int(fname_components[tfevents_idx + 1])
+        except (ValueError, IndexError):
             return False
-        if part != fname_component_part:
+        # Ensure that the file is newer then our start time, and that it was
+        # created from the same hostname.
+        # TODO: we should also check the PID (also contained in the tfevents
+        #     filename). Can we assume that our parent pid is the user process
+        #     that wrote these files?
+        if created_time < int(start_time):
             return False
-    try:
-        created_time = int(fname_components[tfevents_idx + 1])
-    except (ValueError, IndexError):
-        return False
-    # Ensure that the file is newer then our start time, and that it was
-    # created from the same hostname.
-    # TODO: we should also check the PID (also contained in the tfevents
-    #     filename). Can we assume that our parent pid is the user process
-    #     that wrote these files?
-    return created_time >= int(start_time)
+    return True
 
 
 class TBWatcher:
@@ -217,12 +224,13 @@ def _is_our_tfevents_file(self, path: str) -> bool:
         """Check if a path has been modified since launch and contains tfevents."""
         if not path:
             raise ValueError("Path must be a nonempty string")
-        if self._force:
-            return True
         path = self.tf_compat.tf.compat.as_str_any(path)
-        return is_tfevents_file_created_by(
-            path, self._hostname, self._tbwatcher._settings._start_time
-        )
+        if self._force:
+            return is_tfevents_file_created_by(path, None, None)
+        else:
+            return is_tfevents_file_created_by(
+                path, self._hostname, self._tbwatcher._settings._start_time
+            )
 
     def _loader(
         self, save: bool = True, namespace: Optional[str] = None


Test patch: