Trainer not logging to WandB in SageMaker
Author: alexf-a
Pull request: no

- `transformers` version: 4.3.0
- wandb version: 0.10.20
- Platform: SageMaker hosted training with PyTorch estimator. 
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No 

@stas00 @sgugger 

I am using a SageMaker training environment to train `BertForSequenceClassification`. To do this, I'm passing the model into a `Trainer` instance and calling `trainer.train()`.

To train in SageMaker, I am using a PyTorch estimator:

```
estimator = PyTorch(
                    entry_point='train_classifier.py',
                    source_dir='./',
                    role=role,
                    sagemaker_session=sagemaker_session,
                    hyperparameters=hp,
                    subnets=subnets,
                    security_group_ids=sec_groups,
                    framework_version='1.6.0',
                    py_version='py3',
                    instance_count=1,
                    instance_type=instance_type,
                    dependencies=[ '../lib', '../db_conn'],
                    use_spot_instances=False,
                    volume_size=100,
                    #max_wait=max_wait_time_secs
                    )
estimator.fit()
```
I have tried this with different p2 and p3 instances. 

In EC2 or in a SageMaker notebook, this does automated logging of training loss and evaluation loss and metrics to WandB. With the estimator, I get no training logs.

Anything that I manually log to WandB appears in my dashboard. The only info that doesn't show up is whatever used to get logged by the Trainer. 

I tried `os.environ["WANDB_DISALBED"] = "false"` in my training script, no luck. 


Comments:
Are you sure `wandb` is installed in the container you are using for training? There can't be any report if it's not installed and initialized.

Yup, it's installed.  Here is the requirements file that goes into the container

```
boto3==1.16.32
peewee==3.13.3
pandas==1.0.5
torch==1.6.0
numpy==1.18.2
transformers==4.3.0
wandb==0.10.20
botocore==1.19.37
srsly==1.0.2
psycopg2_binary==2.8.5
scikit_learn==0.23.2
uvloop==0.14.0
./prodigy-1.10.5-cp36.cp37.cp38-cp36m.cp37m.cp38-linux_x86_64.whl
https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.3.1/en_core_web_md-2.3.1.tar.gz
```
I also upload my API key from a separate file. 

Also, if I manually call 
`wandb.log({"Some metric":"This will appear"})`

Then it shows up in the run's dashboard. The only information that doesn't show up is the automatic logging that the Trainer is supposed to do. 

Could you share the training script you are using?

I can't share the full script, but it's something like 

```
import wandb
from transformers import (
    AutoConfig,
    AutoModelForSequenceClassification,
    HfArgumentParser,
    Trainer,
    TrainingArguments,
)

class ModelArguments:
#Declare arguments here

class DataTrainingArguments:
#Declare more arguments

parser = HfArgumentParser((ModelArguments, DataTrainingArguments))
model_args, data_args, training_args = parser.parse_args_into_dataclasses()

#Assuming I have the config already
model = AutoModelForSequenceClassification.from_pretrained(
            model_args.model_name_or_path,
            config=config
        )

#train_data, eval_data = get data from a remote host

trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_data,
        eval_dataset=eval_data,
    )

trainer.train()

wandb.log({"Fake Metric":"This will show up"})
```
When I run on SageMaker, I can see from the console logs and the post-training evaluation that it has been training. 



It would be interesting to have the result of 
```
from transformers.integrations import get_available_reporting_integrations
print(get_available_reporting_integrations())
```
to check whether or not `wandb` is listed.

Another thing you can do is force the reporting to wandb with adding `report_to = ["wand"]` in your hyperparameters.

Got this from the integrations print
`['wandb']`
Also, tried to force the reporting to WandB. It's a bit tricky to pass lists into the SageMaker Estimator as a hyperparam, so I just tried to force the reporting in my script

```
    parser = HfArgumentParser((ModelArguments, DataTrainingArguments))
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
    training_args.report_to = ["wandb"]
    data_args.report_to = ["wandb"]
```
Still getting the same result

Curiouser and curiouser. So it is detected an added but doesn't log anything. @borisdayma do you have some idea of why theWandbCallback would not log anything in a training launched on SageMaker?

I'm not sure but now I'm feeling the curiousest!
There is actually some W&B documentation with specific setup instructions for Sagemaker: see [here](https://docs.wandb.ai/integrations/sagemaker)

I'm feeling pretty curious too...already followed the instructions with the secrents.env file. 

Could you add somewhere in the methods of [`WandbCallback`](https://github.com/huggingface/transformers/blob/395ffcd757103ed2ccc888e48d90fd2ccb4d506f/src/transformers/integrations.py#L510) some `print` statements just to see if they are being called at all?

You could just copy that file in your local folder and import from it.

I cloned the repo, added the` print `statements, imported the `WandbCallback` and added it to the `Trainer` callbacks.

I am seeing some `print` statements, but...no logs to WandB dashboard :( 

That's progress!
Can you add some maybe just after the calls to `wandb.log` and maybe also print `wandb.run.id`?

Got some interesting results.

I added

```
    def on_train_begin(self, args, state, control, model=None, **kwargs):
        print("TESTING WANDB: BEGINNING TRAINING")
        wandb.log({"Begining training": "Please appear"})
        ...
```
and ran it locally (in the CLI on a Notebook instance). I got:

`NameError: name 'wandb' is not defined`

If I run this code in the callbacks init function:


```
    def __init__(self):
        print("TESTING WANDB: INITIALIZING CALLBACK")
        has_wandb = is_wandb_available()
        assert has_wandb, "WandbCallback requires wandb to be installed. Run `pip install wandb`."
        if not has_wandb:
            print("NO WEIGHTS AND BIASES!!!!!!!!")
        if has_wandb:
            import wandb

            wandb.ensure_configured()
            if wandb.api.api_key is None:
                has_wandb = False
                logger.warning(
                    "W&B installed but not logged in. Run `wandb login` or set the WANDB_API_KEY env variable."
                )
                self._wandb = None
            else:
                self._wandb = wandb
        self._initialized = False
        # log outputs
        self._log_model = os.getenv("WANDB_LOG_MODEL", "FALSE").upper() in ENV_VARS_TRUE_VALUES.union({"TRUE"})
```
`"NO WEIGHTS AND BIASES!!!!!!!!"` does not print.

If `wandb` gets imported, shouldn't the log statements work? Not sure if this is a separate issue from the SageMaker one. 

Very interesting!
What if you ignore the second check and just set `self._wandb = wandb`

Same error! 

Does [this example](https://github.com/wandb/examples/tree/master/examples/pytorch/pytorch-cifar10-sagemaker) work for you?

Yup

Maybe can you try to log directly something to even before using the `Trainer`?

```python
import wandb
wandb.init()
wandb.log({'val': 1})
```

That way we will see if it comes from your wandb setup in Sagemaker or HF.

I do already do that actually, and it does appear! I successfully log a couple of charts before and after training. 

In your code where you set `self._wandb`, can you try to do something like `self._wandb.log({'test_value': 1})`

This works! 

I have regular training logs and the test_value logged when the callback's int function is like this:

```
    def __init__(self):
        print("TESTING WANDB: INITIALIZING CALLBACK")
        has_wandb = is_wandb_available()
        assert has_wandb, "WandbCallback requires wandb to be installed. Run `pip install wandb`."
        if has_wandb:
            import wandb

            wandb.ensure_configured()
            '''
            if wandb.api.api_key is None:
                has_wandb = False
                logger.warning(
                    "W&B installed but not logged in. Run `wandb login` or set the WANDB_API_KEY env variable."
                )
                self._wandb = None
            else:
            '''
            self._wandb = wandb
            self._wandb.log({'test_value': 1})
        self._initialized = False
        # log outputs
        self._log_model = os.getenv("WANDB_LOG_MODEL", "FALSE").upper() in ENV_VARS_TRUE_VALUES.union({"TRUE"})
```

However, if I do not pass in the modified version of WandbCallback the problem resumes. 

This is very strange...
So just adding the `log` statement makes it work and it stops working if you remove it?

It looks like what makes the difference is commenting out this portion

```
            '''
            if wandb.api.api_key is None:
                has_wandb = False
                logger.warning(
                    "W&B installed but not logged in. Run `wandb login` or set the WANDB_API_KEY env variable."
                )
                self._wandb = None
            else:
            '''
```
And then passing the modified callback in explicitly in the Trainer. 

I believe that the NameError above was because I was calling `wandb.log`, instead of `self._wandb.log`

This should now be fixed on the master branch of `transformers`. Let me know if you still have any issue @alexf-a 

It is still happening! It looks like the WandB API key is not getting properly loaded from the secrets.env. If I run this code, it works

```  
       with open("./secrets.env", "r") as secrets_f:
            wandb_api_key = secrets_f.read().replace('\n', '').split("=")[1]
        os.environ["WANDB_API_KEY"] = wandb_api_key 

```

Hey @alexf-a we'll work on a patch in the next release that handles the sagemaker case.  Until then if you just add this code before you instantiate your trainer it should work.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.