Tracking full integration for fill-in-middle (FIM)
Author: sayakpaul
Pull request: no
Labels: name: WIP

name: contributions-welcome

Given that most code LLMs are trained using the FIM objective [1], I think it makes a lot of sense to work on:

* A training example just like how we have it here: https://github.com/huggingface/transformers/tree/main/examples/pytorch. This can also be turned into a task guide later (example: https://huggingface.co/docs/transformers/tasks/language_modeling).
* A dedicated pipeline so that users can load FIM-trained models easily with `pipeline("fill-in-middle")`. 
* A task page to list all the relevant resources. 

Cc @ArthurZucker since we discussed it internally via Slack. 

**References**

[1] Efficient Training of Language Models to Fill in the Middle, https://arxiv.org/abs/2207.14255.

Comments:
Hi @sayakpaul, I would like to work on this issue!

Awesome!

Pinging @ArthurZucker to check if we should start with pipeline or the example. 

Both are independent, not sure which one will be the easiest 

I will start with the example resource!
Do you recommend I check out any existing resources examples/notebooks on FIM?

Thanks!

Yes!

Hi @sayakpaul, I was busy with some work hence starting this now.

Is there a list of models in HF transformers that support FIM? Or perhaps a faster way of identifying which models support that (via a tag or something similar)?

Cc: @pacman100 

Hi @sayakpaul, is there a set standard way of naming FIM tokens across tokenizers of all models that support this objective? (for example, is it certain that all prefix tokens will be `<fim_prefix>`, etc?)

If not, should I add the prefix, suffix and middle FIM token names as data arguments in the example script (and default to `<fim_prefix>`, `<fim_suffix>`, and `<fim_middle>` if not provided)?

EDIT:
In the latter case, there are two possibilities: 
1. Either the tokenizer doesn't have those tokens (in which case we add these tokens to the tokenizer vocab and resize model embeddings)
2. The user forgot to provide them (same treatment as the first case along with a warning?).

I have created a draft PR for the example. Also have started the work on the pipeline (will create a PR a little later).

You can check CodeLlama which sets the precedent for direct integration, but we can also compare starCoder, which does not have this directly in the tokenizers. Fine with setting new standards as the pipeline PR progresses! 

@ArthurZucker Cool! CodeLlama seems like a great option to start.

Hi, I would like to support on this. @tanaymeh @sayakpaul  please let me know how can I support 

> Hi, I would like to support on this. @tanaymeh @sayakpaul  please let me know how can I support 

Thanks for offering help @codeserra, however I am working on implementing both the example and pipeline and don't need any help at this moment.

Cheers!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@sayakpaul @ArthurZucker Can you please re-open this issue, I am still very much working on this, though a little swamped by exams this week!

Not stale.

Not stale.