metadata
language:
- en
tags:
- text2text-generation
widget:
- text: >-
Please answer to the following question. Who is going to be the next
Ballon d'or?
example_title: Question Answering
- text: >-
Q: Can Geoffrey Hinton have a conversation with George Washington? Give
the rationale before answering.
example_title: Logical reasoning
- text: >-
Please answer the following question. What is the boiling point of
Nitrogen?
example_title: Scientific knowledge
- text: >-
Answer the following yes/no question. Can you write a whole Haiku in a
single tweet?
example_title: Yes/no question
- text: >-
Answer the following yes/no question by reasoning step-by-step. Can you
write a whole Haiku in a single tweet?
example_title: Reasoning task
- text: 'Q: ( False or not False or False ) is? A: Let''s think step by step'
example_title: Boolean Expressions
- text: >-
The square root of x is the cube root of y. What is y to the power of 2,
if x = 4?
example_title: Math reasoning
- text: >-
Premise: At my age you will probably have learned one lesson.
Hypothesis: It's not certain how many lessons you'll learn by your
thirties. Does the premise entail the hypothesis?
example_title: Premise and hypothesis
datasets:
- Muennighoff/flan
- Open-Orca/SlimOrca-Dedup
- garage-bAInd/Open-Platypus
- Weyaxi/HelpSteer-filtered
- GAIR/lima
license: mit
Model Card for the test-version of instructionBERT for Bertology
A minimalistic instruction model with an already good analysed and pretrained encoder like BERT. So we can research the Bertology with instruction-tuned models, look at the attention and investigate what happens to BERT embeddings during fine-tuning.
The trainings code is released at the instructionBERT repository. We used the Huggingface API for warm-starting BertGeneration with Encoder-Decoder-Models for this purpose.
Run the model with a longer output
from transformers import AutoTokenizer, EncoderDecoderModel
# load the fine-tuned seq2seq model and corresponding tokenizer
model_name = "Bachstelze/instructionBERTtest"
model = EncoderDecoderModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
input = "Write a poem about love, peace and pancake."
input_ids = tokenizer(input, return_tensors="pt").input_ids
output_ids = model.generate(input_ids, max_new_tokens=200)
print(tokenizer.decode(output_ids[0]))
Training parameters
- base model: "bert-base-cased"
- test subset of the Muennighoff/flan dataset
- trained for 0.97 epochs
- batch size of 14
- 10000 warm-up steps
- learning rate of 0.00005
Purpose of instructionBERT
InstructionBERT is intended for research purposes. The model-generated text should be treated as a starting point rather than a definitive solution for potential use cases. Users should be cautious when employing these models in their applications.