Few-Shot Learning Attempt Failure on HF Inference API with GPT-J
Author: dauletbeck
Pull request: no

## Environment info

- `transformers` version: 4.6.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.10
- PyTorch version (GPU?): 1.8.1
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: Unknown

@patil-suraj , @patrickvonplaten , @sgugger


## Information

Model I am using: GPT-J 6B, GPT-NEO 2.7B

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ o ] my own modified scripts: (give details below)
I am trying to make a model generate a text based on a given prompt with examples (Few-Shot Learning). Example of the prompt: 
```
prompt_tweet = """Generate tweet text from a key word:

key: markets
tweet: Take feedback from nature and markets, not from people
###
key: children
tweet: Maybe we die so we can come back as children.
###
key: startups
tweet: Startups should not worry about how to put out fires, they should worry about how to start them.
###
key: NLP
tweet: """
```
However, the model seems to struggle with properly identifying 1) Response Length, 2) End/Stop sequence (i.e. "###" in the example above). How can I achieve results like [here](https://huggingface.co/blog/few-shot-learning-gpt-neo-and-inference-api)? I am not actually sure if I am using the correct parameter names, even though I have looked into everything in the API documentation, there is no indication of that. The parameters I am passing to my model are indicated under the "##To reproduce" section.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ o ] my own task or dataset: (give details below)

I am trying to develop a way for the model to generate text based on Few-Shot Learning, something like [here](https://huggingface.co/blog/few-shot-learning-gpt-neo-and-inference-api). However, I am not sure how to declare an end sequence properly, since I was not able to find any indication of that in the API documentation.
## To reproduce
```import json

import requests

API_URL = "https://api-inference.huggingface.co/models/EleutherAI/gpt-j-6B" #gpt-j default
API_URL1 = "https://api-inference.huggingface.co/models/EleutherAI/gpt-neo-2.7B" #gpt-neo-2.7B
headers = {"Authorization": f"Bearer {API_TOKEN}"}

def query(payload):
    data = json.dumps(payload)
    response = requests.request("POST", API_URL, headers=headers, data=data)
    return json.loads(response.content.decode("utf-8"))

options = {"use_gpu": False,
           "use_cache": False}

parameters = {
              "temperature": 0.6,
              #"top_k":30,
              "repetition_penalty": None,
              "max_new_tokens": None,
              "max_time": None,
              "top_p": 1.0,
              "return_full_text": False,
              "num_return_sequences": 1,
              #"min_length": 100,
              "max_length": len(prompt_tweet),
              "end_sequence": "###"
             }

data = query({"inputs": prompt_tweet, "parameters": parameters, "options": options})
print(data[0]['generated_text'])
```

The code above is pretty much copy-pasted from the website, apart from parameters values.

Steps to reproduce the behavior:

1. Make an API call (Model: GPT-J)
2. Pass in the parameters indicated above
3. Call the query function

## Expected behavior

The model is supposed to generate sensible results more or less consistently when using Few-Shot Learning Approach, I know that by using other API tools I found (see - https://hub.getneuro.ai/model/nlp/gpt-j-6B-text-generation ; https://nlpcloud.io/nlp-text-generation-api-gpt-neo-gpt-j-gpt-3-alternatives.html)


Comments:
I am sorry if I have done some formatting wrong, please let me know if I should improve something.

Gently pinging @Narsil here

Hi @kaisardauletbek ,

For formatting you need to use 3 backticks ` ``` ` to get multiline formatting.

As for your request, the `end_sequence` was not properly respected by this model (it's slightly custom because of memory requirements.

This should be adjusted now.

That being said, your example does NOT use the `end_sequence` within it's prompt meaning the model is unlikely to use it too.
You probably need to use it within your prompts.

Hope this helps,

Cheers,
Nicolas

Hello!

Reformatted. Thanks.

Can you please elaborate on "example does not use the ```end_sequence``` within it's prompt? Do you mean that it should be ```\n###```? If so, then I have tried that too; if not, can you please clarify how to use it within the prompt?

Thanks,
KD

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@kaisardauletbek sorry about late answer, notifications on github are shaky sometimes. (Don't hesitate to ping if it happens a again).

What I meant is that yes.

`\n###` is a single token with that model, meaning `###` as end_sequence is just ignored (because the ids doesn't match what's used in the prompt, namely `\n###`. Using this instead should yield better resutls.

Sorry if my answer was not very clear.




This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.