parm:logging_dir not works when 
Author: chenjinhongv
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.16.2
- Platform:linux
- Python version:3.7.11
- PyTorch version (GPU?):1.10.2 GPU
- Tensorflow version (GPU?):no tensorflow
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:

## Information

Model I am using (Bert, XLNet ...):unicamp-dl/translation-en-pt-t5

The problem arises when using:
* [ ] the official example scripts: (give details below)
`from transformers import BertForSequenceClassification, Trainer, TrainingArguments

model = BertForSequenceClassification.from_pretrained("bert-large-uncased")

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # total # of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
)

trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=test_dataset            # evaluation dataset
)`
* [ ] my own modified scripts: (give details below)
`
model_checkpoint = "unicamp-dl/translation-en-pt-t5"
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)
args = Seq2SeqTrainingArguments(
    output_dir=f"{model_name}-finetuned-{source_lang}-to-{target_lang}-{time_stamp}",  
    overwrite_output_dir=True,  
    evaluation_strategy="epoch",
    prediction_loss_only=False,  
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,  # batch size per device during training
    per_device_eval_batch_size=batch_size,   # batch size for evaluation
    weight_decay=0.01,
    save_steps=3000,  
    eval_steps=10000,  
    logging_dir='./logs',
    logging_steps=500,  
    save_total_limit=3,  
    num_train_epochs=10,
    predict_with_generate=True,
    fp16=True,  
    push_to_hub=False
)

trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["valid"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)

trainer.train()
`

I could not found the dir logs which should be found in same dir of the script above 

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
![image](https://user-images.githubusercontent.com/44344942/156742529-e062727f-c60c-4d22-8134-7325ee0bd29d.png)


@sgugger thanks

Comments:
How do you launch the script? In particular if you launch it from another directory, the logging dir will appear there.

thanks very much，I found it is because I launch the script with Environment without Tensorflow installed（by the way, I get no warning about this during the process, haha）

I have same problem.
When Run with 1 GPU or no GPU environment, That option works fine. but with multi GPU it not works.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.