NotebookProgressCallback doesn't work in databricks notebooks properly--should either be fixed or removed from trainer automatically if it is a databricks runtime
Author: Ashvio
Pull request: no

https://github.com/huggingface/transformers/blob/71e602725b90f63f404109bae9f72cbdf755477b/src/transformers/utils/notebook.py#L269

Hey all,
Using databricks for training, the default Trainer behavior will automatically add NotebookProgressCallback for databricks notebooks, but the databricks UI currently does not display the output properly. It just prints a bunch of text saying `<IPython.core.display.HTML object>` over and over. This is likely an issue on Databricks' end, so I recommend not adding this callback if the transformers library can detect it is a databricks runtime not a jupyter/google collab notebook. I think there should also be an easier way to delete specific callbacks--it took a long time to trace this issue and reading source code to figure out what the source cause is. I am circumventing the issue for now by popping the callback from the trainer callback handlers list but that is not a good pattern.

Thanks!

Comments:
Do you know which test we could do to easily detect if we are in a `Databrick` runtime?

> Do you know which test we could do to easily detect if we are in a `Databrick` runtime?

From stackoverflow:

```
def isRunningInDatabricks(): Boolean = 
  sys.env.contains("DATABRICKS_RUNTIME_VERSION")
```



`sys.env` is not something that exists. Did you mean `os.environ`?

Could you try if the PR mentioned above does solve your problem?

Hi, just want to add, since I experienced the same issue in the past.

I believe the reason why the HTML produced by `NotebookProgressCallback` is not displayed properly is because Databricks (with runtime version prior to 11.0) is not using IPython kernel to execute the Python code.

There was a guide how to set Databricks to use IPython kernel. And in my experience, when this is set, the evaluation result table produced by `NotebookProgressCallback` is displayed properly. 

https://web.archive.org/web/20211227103927/https://docs.microsoft.com/en-us/azure/databricks/notebooks/ipython-kernel

Most users, however, I believe will use the default setting i.e. not overriding Databricks default setting to specifically use IPython kernel. Therefore, the changes in this [commit](https://github.com/huggingface/transformers/pull/17496) looks good.

However, in the most recent Databricks runtime version 11.0, IPython kernel is now the default Python code execution engine. Therefore,  the HTML produced by `NotebookProgressCallback` I believe can be displayed **correctly** by default in Databricks runtime 11.x

https://docs.microsoft.com/en-us/azure/databricks/notebooks/ipython-kernel

I suggest, in addition to checking if this environment variable `DATABRICKS_RUNTIME_VERSION` is set, we should also check the version. If the version is 11.x, I believe it is ok to use the `NotebookProgressCallback`. It can show the table HTML output properly in my test.

![image](https://user-images.githubusercontent.com/5300554/176932865-686503c9-fb4d-409e-85bf-cfa749f7796b.png)



If you want to make a PR with the adjustment, I'll be happy to look at it!

> If you want to make a PR with the adjustment, I'll be happy to look at it!

https://github.com/huggingface/transformers/pull/17988

Thanks