Fix(43240): pass kwargs to nn.functional.cross_entropy
Author: jasiecky
Pull request: yes

# What does this PR do?

The problem to be solved is the issue https://github.com/huggingface/transformers/issues/43240. This PR implements passing weight and label_smoothing parameters of nn.functional.cross_entropy in fixed_cross_entropy function.

Fixes #43240 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@iamsernine @ArthurZucker @stas00 @cyyever 

Comments:
Thank you for adding the tests. I still don't understand what you're trying to solve with this PR. 

Your PR description:
> This PR adds validation for keyword arguments passed to cross_entropy and ensures consistent loss scaling by controlling the reduction mode when num_items_in_batch is provided.

The 2nd part is invalid, the pre-PR code already does that.

For the first part, what `kwargs` do you need to pass for your workload? Your tests exercise the 2 keys that have been ignored, but do you actually need them? It's been a long time since this function was added - perhaps `fixed_` implies that it does a limit scope of things. I'm not sure. The trainer for example performs label_smoothing here https://github.com/huggingface/transformers/blob/0a5d574dd0d0dadfbbc981a572bd585284c2ed59/src/transformers/trainer.py#L708-L709



Thinking more about it and stepping away from this particular PR, one of the remaining issues in the HF Transformers API is that some keys in `kwargs` are silently dropped in some of the APIs. For example if you call `from_config` and set `config.attn_implementation` it will be silently ignored - it should assert and tell the user to pass `attn_implementation` as its own `kwargs` key in `from_config` and not part of the config object.

So what I suggest is that if you don't have a particular problem to solve in this PR, it can be made useful by asserting when unexpected `kwargs` are passed - those must not be silently dropped. The user needs to know when they are not using the API correctly.

The decision of whether `fixed_cross_entropy` should support `weight` and `label_smoothing` kwargs I will leave to the current maintainers.

> Thinking more about it and stepping away from this particular PR, one of the remaining issues in the HF Transformers API is that some keys in `kwargs` are silently dropped in some of the APIs. For example if you call `from_config` and set `config.attn_implementation` it will be silently ignored - it should assert and tell the user to pass `attn_implementation` as its own `kwargs` key in `from_config` and not part of the config object.
> 
> So what I suggest is that if you don't have a particular problem to solve in this PR, it can be made useful by asserting when unexpected `kwargs` are passed - those must not be silently dropped. The user needs to know when they are not using the API correctly.
> 
> The decision of whether `fixed_cross_entropy` should support `weight` and `label_smoothing` kwargs I will leave to the current maintainers.

The problem to be solved is the issue [43240](https://github.com/huggingface/transformers/issues/43240). The thing is that currently we are not able to pass kwargs into nn.functional.cross_entropy so usage of weight and label_smoothing is impossible. If you think it would be a better solution I might change kwargs to these two parameters and pass them into the mentioned function.

That's helpful. Then that should be the description of the PR.

And you have a competitor here: https://github.com/huggingface/transformers/pull/43254

> That's helpful. Then that should be the description of the PR.
> 
> And you have a competitor here: #43254

I updated the code and the description, ready for review.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43251&sha=6e287b