--- language: - en license: mit library_name: transformers tags: - batch prompting - batch - BatchPrompt - BatchPrompting - GLUE - Llama - fine-tuned - Llama3 - Llama-3-8B-Instruct datasets: - achandlr/BatchPrompting metrics: - accuracy pipeline_tag: question-answering --- # Model Card for Model ID This model is a fine-tuned version of Llama-3-8B-Instruct on the BatchPrompting dataset, which spans 13 diverse NLP tasks. The model has been fine-tuned to effectively perform batch prompting - answering multiple questions concatenated into a single prompt in one inference pass. ## Model Details This model is a fine-tuned version of Llama-3-8B-Instruct on the BatchPrompting dataset, which spans 13 diverse NLP tasks. The model has been fine-tuned to effectively perform batch prompting - answering multiple questions concatenated into a single prompt in one inference pass. ### Model Description - **Developed by:** Alex Chandler, Sebastian Joseph - **Model type:** Large Language Model (Llama-3 variant - **Language(s) (NLP):** English - **License:** MIT - **Finetuned from model [optional]:** Llama-3-8B-Instruct ### Model Sources [optional] - **Repository:** Forthcoming - **Paper:** Forthcoming - **Demo:** Forthcoming ## Uses ### How to Use Use with transformers See the snippet below for usage with Transformers: ```python import transformers import torch model_id = "achandlr/Llama-3-8B-Instruct-BatchPromptQA" # Load the model pipeline pipeline = transformers.pipeline("text-generation", model=model_id) # Generate text using the pipeline generated_text = pipeline("Hey how are you doing today?") print(generated_text) ``` ### Direct Use The model can be used for efficient question-answering on a variety of NLP tasks by concatenating multiple questions into a single prompt. It demonstrates strong generalization to unseen tasks and maintains performance with larger batch sizes compared to the non-fine-tuned model. ### Out-of-Scope Use The model should not be used for tasks that may cause harm or for generating factually incorrect or biased content. Caution should be exercised if using the model for high-stakes decision making. ## Bias, Risks, and Limitations The model may exhibit biases present in its pretraining data or the BatchPrompting dataset. It has not been extensively tested for fairness or potential misuse. Performance may degrade on out-of-distribution examples or tasks very dissimilar to the training data. ### Recommendations Users should be made aware of the model's potential limitations and biases. The model's outputs should be carefully monitored, especially when used for sensitive applications. More testing is needed to fully characterize its capabilities and shortcomings. ## How to Get Started with the Model Use the code below to get started with the model. [More Information Needed] ## Training Details ### Training Data The model was fine-tuned on our BatchPrompting dataset consisting of 13 NLP tasks: - **GLUE Benchmark Tasks**: A collection of datasets used for evaluating the performance of models on a variety of natural language understanding tasks. - **Mathematical Reasoning Datasets**: - **GSM8K**: Focuses on numerical and logical reasoning challenges. - **GSM8K-Hard**: Contains more complex problems from the GSM8K dataset. - **CommonsenseQA**: Tests the model's commonsense reasoning ability through multiple-choice question answering. - **RACE Reading Comprehension Dataset**: Consists of passages and questions designed to assess reading comprehension, derived from English exams. ### Training Procedure The model was fine-tuned using the LoRA method. #### Training Hyperparameters - **Training regime:** Forthcoming #### Speeds, Sizes, Times [optional] Forthcoming [More Information Needed] ## Evaluation Testing Data, Factors & Metrics Evaluation was performed on tasks that were excluded from the training run. Key metrics included accuracy and BatchPrompt error rate (failure to answer a question or conform to the specified format). A table of our results is forthcoming. ### Testing Data, Factors & Metrics Forthcoming #### Testing Data Forthcoming [More Information Needed] #### Metrics Forthcoming [More Information Needed] ### Results Forthcoming [More Information Needed] #### Summary Forthcoming ## Model Examination [optional] Forthcoming [More Information Needed] ### Model Architecture and Objective [More Information Needed] ### Compute Infrastructure [More Information Needed] #### Hardware [More Information Needed] #### Software [More Information Needed] ## Citation Forthcoming **BibTeX:** Forthcoming **APA:** [More Information Needed] ## Glossary [optional] [More Information Needed] ## More Information [optional] [More Information Needed] ## Model Card Authors [optional] [More Information Needed] ## Model Card Contact [More Information Needed]