--- license: cc-by-sa-4.0 language: - en library_name: transformers pipeline_tag: text2text-generation tags: - text-generation-inference base_model: - meta-llama/Meta-Llama-3-8B-Instruct new_version: FrancescoPeriti/Llama3Dictionary-merge --- # Llama3Dictionary ```FrancescoPeriti/Llama3Dictionary``` is a fine-tuned version of the ```meta-llama/Meta-Llama-3-8B-Instruct```. Thus, to use it, visit the AI at Meta website, accept the Meta License, and submit the [form](https://llama.meta.com/llama-downloads/). You will need to login with your hugginface token (```[HF-TOKEN]```, in the following). ### Model Description This model is fine-tuned on English datasets of sense definitions. Given a target word and a usage example, the model generates a sense definition for the target word in-context. You can find more details in the paper [Automatically Generated Definitions and their utility for Modeling Word Meaning](https://aclanthology.org/2024.emnlp-main.776/) by Francesco Periti, David Alfter, Nina Tahmasebi. The repository of our project is [https://github.com/FrancescoPeriti/LlamaDictionary](https://github.com/FrancescoPeriti/LlamaDictionary). ## Uses The model is designed for research purposes and is conceived to work like a dictionary. However, given a word and an example usage, users don't choose from a list of definitions (as in a traditional dictionary); instead, the model directly provides the sense definition for the word in-context. ## Bias, Risks, and Limitations The fine-tuning datasets were limited to English, and generated definitions may reflect biases and stereotypes inherent in the underlying language model. ## How to Get Started with the Model ```python import torch import warnings from peft import PeftModel # parameter-efficient fine-tuning from datasets import Dataset from huggingface_hub import login from typing import (Literal, Sequence,TypedDict) from transformers import AutoTokenizer, AutoModelForCausalLM login([HF-TOKEN]) # e.g., hf_aGPI...ELal model_name = "meta-llama/Meta-Llama-3-8B-Instruct" # chat model ft_model_name = "FrancescoPeriti/Llama3Dictionary" # fine-tuned model # load models chat_model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto') lama3dictionary = PeftModel.from_pretrained(chat_model, ft_model_name) lama3dictionary.eval() # load tokenizer tokenizer = AutoTokenizer.from_pretrained( model_name, padding_side="left", add_eos_token=True, add_bos_token=True, ) tokenizer.pad_token = tokenizer.eos_token # end of sequence for stop condition eos_tokens = [tokenizer.encode(token, add_special_tokens=False)[0] for token in [';', ' ;', '.', ' .']] eos_tokens.append(tokenizer.eos_token_id) # chat format Role = Literal["system", "user"] class Message(TypedDict): role: Role content: str Dialog = Sequence[Message] # load dataset examples = [{'target': 'jam', 'example': 'The traffic jam on the highway made everyone late for work.'}, {'target': 'jam', 'example': 'I spread a generous layer of strawberry jam on my toast this morning'}] dataset = Dataset.from_list(examples) # apply template def apply_chat_template(tokenizer, dataset): system_message = "You are a lexicographer familiar with providing concise definitions of word meanings." template = 'Please provide a concise definition for the meaning of the word "{}" in the following sentence: {}' def apply_chat_template_func(record): dialog: Dialog = (Message(role='system', content=system_message), Message(role='user', content=template.format(record['target'], record['example']))) prompt = tokenizer.decode(tokenizer.apply_chat_template(dialog, add_generation_prompt=True)) return {'text': prompt} return dataset.map(apply_chat_template_func) dataset = apply_chat_template(tokenizer, dataset) # tokenization max_length = 512 def formatting_func(record): return record['text'] def tokenization(dataset): result = tokenizer(formatting_func(dataset), truncation=True, max_length=max_length, padding="max_length", add_special_tokens=False) return result tokenized_dataset = dataset.map(tokenization) # definition generation batch_size = 32 max_time = 4.5 # sec sense_definitions = list() with torch.no_grad(): for i in range(0, len(tokenized_dataset), batch_size): batch = tokenized_dataset[i:i + batch_size] model_input = dict() for k in ['input_ids', 'attention_mask']: model_input[k] = torch.tensor(batch[k]).to('cuda') output_ids = lama3dictionary.generate(**model_input, max_length = max_length, forced_eos_token_id = eos_tokens, max_time = max_time * batch_size, eos_token_id = eos_tokens, temperature = 0.00001, pad_token_id = tokenizer.eos_token_id) answers = tokenizer.batch_decode(output_ids, skip_special_tokens=True) for j, answer in enumerate(answers): answer = answer.split('\n')[-1].strip(" .,;:") if len(answer) == 0: warnings.warn("Something went wrong. The input example might be too long; try reducing it.") sense_definitions.append(answer.replace('\n', ' ') + '\n') # output dataset = dataset.add_column('definition', sense_definitions) for row in dataset: print(f"Target: {row['target']}\nExample: {row['example']}\nSense definition: {row['definition']}") ``` ## Citation Francesco Periti, David Alfter, and Nina Tahmasebi. 2024. [Automatically Generated Definitions and their utility for Modeling Word Meaning](https://aclanthology.org/2024.emnlp-main.776/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14008–14026, Miami, Florida, USA. Association for Computational Linguistics. **BibTeX:** ``` @inproceedings{periti2024automatically, title = {{Automatically Generated Definitions and their utility for Modeling Word Meaning}}, author = "Periti, Francesco and Alfter, David and Tahmasebi, Nina", editor = "Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung", booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing", month = nov, year = "2024", address = "Miami, Florida, USA", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.emnlp-main.776", pages = "14008--14026", abstract = "Modeling lexical semantics is a challenging task, often suffering from interpretability pitfalls. In this paper, we delve into the generation of dictionary-like sense definitions and explore their utility for modeling word meaning. We fine-tuned two Llama models and include an existing T5-based model in our evaluation. Firstly, we evaluate the quality of the generated definitions on existing English benchmarks, setting new state-of-the-art results for the Definition Generation task. Next, we explore the use of definitions generated by our models as intermediate representations subsequently encoded as sentence embeddings. We evaluate this approach on lexical semantics tasks such as the Word-in-Context, Word Sense Induction, and Lexical Semantic Change, setting new state-of-the-art results in all three tasks when compared to unsupervised baselines.", } ```