|
--- |
|
library_name: peft |
|
base_model: NousResearch/Llama-2-7b-hf |
|
license: llama2 |
|
datasets: |
|
- DehydratedWater42/semantic_relations_extraction |
|
language: |
|
- en |
|
inference: false |
|
tags: |
|
- math |
|
- semantic |
|
- extraction |
|
- graph |
|
- relations |
|
- science |
|
- synthetic |
|
--- |
|
|
|
# SeELLama (Semantic Extraction LLama) |
|
Model is based on LLama2-7b and fine-tuned with the `DehydratedWater42/semantic_relations_extraction` dataset. |
|
|
|
The purpose of this model is to extract semantic relations from text in a structured way. |
|
|
|
#### Simplified Example: |
|
- **Initial Text**: "While there is beautiful weather outside the building, from the window we can see a car. And what's the most annoying, pigeons love to sit on that car." |
|
- **Entities**: ["pigeon", "car", "building"] |
|
- **Relations between entities**: {"pigeon -> car": "pigeon sits on the car", "car -> building": "car is parked outside the building"} |
|
|
|
**Note:** The text example above is **too short** for the actual model; please use **at least 500-token text** segments for extraction to avoid hallucinations. |
|
|
|
### This is just adapter for `NousResearch/Llama-2-7b-hf` |
|
- **Get SeELLama as Safetensors:** [DehydratedWater42/SeELLama](https://huggingface.co/DehydratedWater42/SeELLama) |
|
- **Get SeELLama as GGUF:** [DehydratedWater42/SeELLama-GGUF](https://huggingface.co/DehydratedWater42/SeELLama-GGUF) |
|
|
|
## How to use it: |
|
### Template: |
|
```text |
|
Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format. |
|
### Text Part to Extract From: |
|
<<your_text_for_extraction>> |
|
### Extracted Relations: |
|
{ |
|
"section_description": |
|
``` |
|
The `JSON` opening is not necessary but it improves stability. Remember to use double `{{` instead of singular '{' if you are using LangChain prompts with fstring formatting. |
|
|
|
## Fine-tuning code/settings |
|
|
|
```python |
|
tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-hf", trust_remote_code=True) |
|
tokenizer.pad_token = tokenizer.eos_token |
|
tokenizer.padding_side = "right" |
|
|
|
torch.backends.cuda.matmul.allow_tf32 = True |
|
torch.backends.cudnn.allow_tf32 = True |
|
|
|
bnb_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_use_double_quant=True, |
|
bnb_4bit_quant_type="nf4", |
|
bnb_4bit_compute_dtype=torch.bfloat16, |
|
) |
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
"NousResearch/Llama-2-7b-hf", |
|
quantization_config=bnb_config, |
|
use_cache=False, |
|
use_flash_attention_2=False, |
|
device_map="auto", |
|
) |
|
model.config.pretraining_tp = 1 |
|
|
|
peft_params = LoraConfig( |
|
lora_alpha=16, |
|
lora_dropout=0.1, |
|
r=64, |
|
bias="none", |
|
task_type="CAUSAL_LM", |
|
) |
|
|
|
training_params = TrainingArguments( |
|
output_dir="./results", |
|
num_train_epochs=3, |
|
per_device_train_batch_size=1, |
|
gradient_accumulation_steps=1, |
|
optim="paged_adamw_32bit", |
|
save_steps=100, |
|
logging_steps=1, |
|
learning_rate=2e-4, |
|
weight_decay=0.001, |
|
bf16=True, |
|
fp16=False, |
|
tf32=True, |
|
max_grad_norm=0.3, |
|
max_steps=-1, |
|
warmup_ratio=0.03, |
|
group_by_length=True, |
|
lr_scheduler_type="constant", |
|
report_to="mlflow", |
|
run_name="semantic-extraction-llama2-7b" |
|
) |
|
|
|
trainer = SFTTrainer( |
|
model=model, |
|
train_dataset=formated_dataset, |
|
peft_config=peft_params, |
|
dataset_text_field="text", |
|
max_seq_length=int(1024 * 2.5), # 2560 |
|
tokenizer=tokenizer, |
|
args=training_params, |
|
packing=True, |
|
) |
|
``` |
|
|
|
### Example: |
|
Extracting information from Minecraft Wiki. |
|
|
|
#### Initial template |
|
```text |
|
Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format. |
|
### Text Part to Extract From: |
|
Players in Survival, Adventure, or Spectator have access to the 2×2 crafting grid from their inventory screen. Small crafting recipes that are at most 2×2 can be made there. These include wooden planks, sticks, crafting tables, torches and some shapeless recipes. To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it. This brings up an interface with a larger, 3×3 crafting grid, which the player can use to craft any crafting recipe in the game. There is also a recipe book where the player can access all the crafting recipes. |
|
|
|
Some recipes do not require their ingredients to be arranged in a specific way on the crafting grid. These are commonly known as shapeless recipes. For example, players may craft a fermented spider eye by placing its ingredients anywhere within the grid or like a flint and steel can be crafted in any orientation. |
|
|
|
On the contrary, many recipes must have their ingredients placed in the correct relative positions on the crafting grid. These are commonly known as shaped recipes. Ingredients in shaped recipes can be ‘moved’ up, down, left, or right. They can also be flipped side-ways. For example, a 3×1 recipe, such as bread, can be made using the top, middle, or bottom row of the 3×3 grid, and a bow may be made with the strings placed on the left instead of on the right. |
|
|
|
There are recipes that may not be moved or mirrored in this way. These are commonly known as fixed recipes. For example, dyes in banner recipes - only available in Bedrock Edition - must be specifically placed to achieve the desired pattern. Fixed recipes can be added by data packs, add-ons, or mods. |
|
### Extracted Relations: |
|
{ |
|
"section_description": |
|
``` |
|
|
|
|
|
#### Result (about game): |
|
```text |
|
Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format. |
|
### Text Part to Extract From: |
|
Players in Survival, Adventure, or Spectator have access to the 2×2 crafting grid from their inventory screen. Small crafting recipes that are at most 2×2 can be made there. These include wooden planks, sticks, crafting tables, torches and some shapeless recipes. To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it. This brings up an interface with a larger, 3×3 crafting grid, which the player can use to craft any crafting recipe in the game. There is also a recipe book where the player can access all the crafting recipes. |
|
|
|
Some recipes do not require their ingredients to be arranged in a specific way on the crafting grid. These are commonly known as shapeless recipes. For example, players may craft a fermented spider eye by placing its ingredients anywhere within the grid or like a flint and steel can be crafted in any orientation. |
|
|
|
On the contrary, many recipes must have their ingredients placed in the correct relative positions on the crafting grid. These are commonly known as shaped recipes. Ingredients in shaped recipes can be ‘moved’ up, down, left, or right. They can also be flipped side-ways. For example, a 3×1 recipe, such as bread, can be made using the top, middle, or bottom row of the 3×3 grid, and a bow may be made with the strings placed on the left instead of on the right. |
|
|
|
There are recipes that may not be moved or mirrored in this way. These are commonly known as fixed recipes. For example, dyes in banner recipes - only available in Bedrock Edition - must be specifically placed to achieve the desired pattern. Fixed recipes can be added by data packs, add-ons, or mods. |
|
### Extracted Relations: |
|
{ |
|
"section_description": "In Minecraft, players can craft items using the 2x2 crafting grid. There are some fixed recipes that require the player to place their ingredients in a specific position.", |
|
"list_of_entities": [ |
|
"crafting recipes", |
|
"player", |
|
"fixed recipes", |
|
"2x2 crafting grid", |
|
"survival, adventure, or spectator players", |
|
"wooden planks", |
|
"torches", |
|
"shaped recipes", |
|
"small crafting recipes", |
|
"shapeless recipes", |
|
"wooden planks, sticks, crafting tables, torches", |
|
"flint and steel" |
|
], |
|
"relations": [ |
|
{ |
|
"description": "Players have access to the 2x2 crafting grid from their inventory screen.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"player" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Small crafting recipes that are at most 2x2 can be made using the 2x2 crafting grid.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"survival, adventure, or spectator players" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"wooden planks" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"crafting tables" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Crafting tables allow players to make bigger crafting recipes that are not possible with the 2x2 crafting grid.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"torches", |
|
"flint and steel", |
|
"wooden planks" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Crafting tables are used to craft items using the 2x2 crafting grid.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"wooden planks, sticks, crafting tables, torches" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "There are some fixed recipes that require the player to place their ingredients in a specific position.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"fixed recipes" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "The recipes do not require their ingredients to be arranged in a specific way on the crafting grid.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"shapeless recipes" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Ingredients in shapeless recipes can be ‘moved’ up, down, left, or right.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"shaped recipes" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Ingredients in shaped recipes can be flipped side-ways.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"fixed recipes" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "There are some fixed recipes that require the player to place their ingredients in a specific position.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"fixed recipes" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "The recipes can be added by data packs, add-ons, or mods.", |
|
"source_entities": [ |
|
"crafting recipes" |
|
], |
|
"target_entities": [ |
|
"fixed recipes" |
|
], |
|
"strength": "strong" |
|
} |
|
] |
|
} |
|
``` |
|
|
|
### Smaller example (About crafting) |
|
```text |
|
Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format. |
|
### Text Part to Extract From: |
|
Players in Survival, Adventure, or Spectator have access to the 2×2 crafting grid from their inventory screen. Small crafting recipes that are at most 2×2 can be made there. These include wooden planks, sticks, crafting tables, torches and some shapeless recipes. To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it. This brings up an interface with a larger, 3×3 crafting grid, which the player can use to craft any crafting recipe in the game. There is also a recipe book where the player can access all the crafting recipes. |
|
### Extracted Relations: |
|
{ |
|
"section_description": "Survival, Adventure, and Spectator players have access to the 2×2 crafting grid. However, to craft items using a bigger grid, you need to create a crafting table with wooden planks, place it in the world, and press use while facing it.", |
|
"list_of_entities": [ |
|
"Survival", |
|
"wooden planks", |
|
"Spectator", |
|
"Adventure", |
|
"crafting grid" |
|
], |
|
"relations": [ |
|
{ |
|
"description": "Survival players can access the 2×2 crafting grid.", |
|
"source_entities": [ |
|
"Survival" |
|
], |
|
"target_entities": [ |
|
"crafting grid" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Adventure and Spectator players can also access the 2×2 crafting grid.", |
|
"source_entities": [ |
|
"Adventure" |
|
], |
|
"target_entities": [ |
|
"crafting grid" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "To craft items using a bigger grid, you need to create a crafting table with wooden planks.", |
|
"source_entities": [ |
|
"Spectator" |
|
], |
|
"target_entities": [ |
|
"crafting grid" |
|
], |
|
"strength": "strong" |
|
} |
|
] |
|
} |
|
``` |
|
|
|
### Longer example (dirt block) |
|
```text |
|
Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format. |
|
### Text Part to Extract From: |
|
Usage |
|
Using bone meal on grass blocks causes short grass, tall grass,[Java Edition only] ferns,[Bedrock Edition only] and flowers to grow. |
|
|
|
Passive mobs tend to wander toward grass blocks. They also wander toward light. |
|
|
|
When a sheep eats a grass block, the block becomes dirt, and a sheared sheep regrows its wool. Baby sheep graze grass much more often than adults and mature 1 minute faster when grazing. |
|
|
|
Tilling a grass block with a hoe converts it to a farmland block. |
|
|
|
Dirt paths can be created by using any type of shovel on the side or top of a grass block with air above it. The shovel loses 1 durability for each dirt path block created. |
|
|
|
Death |
|
Grass dies and changes to dirt a random time (when a random tick lands on the block) after it has been covered by any opaque blocks. Transparent blocks can kill grass in a similar manner if they cause the light level above the grass block to be four or below (like water does), and the surrounding area is not otherwise sufficiently illuminated. |
|
|
|
Spawning |
|
In Java Edition, animals occasionally spawn on grass blocks that have light level 9 or brighter in the space directly above. This is quite rare and requires that there be few other animals nearby or in the spawn chunks. Most animals are instead created along with the terrain. |
|
|
|
In Bedrock Edition, most animals require surface grass blocks (grass blocks with no solid block anywhere above them at the same x, z coordinate) to spawn. Animals regularly spawn in most Overworld biomes on surface grass blocks that have light level 7 or brighter in the space directly above. |
|
|
|
Snowy grass block |
|
|
|
A snowy grass block with snow on top of it. |
|
Grass blocks turn into a snowy grass block if snow, powder snow or a snow block is placed on top of the block. Removing the snow layer turns the block into a normal grass block. |
|
|
|
Composting |
|
In Bedrock Edition, placing grass blocks into a composter has a 30% chance of raising the compost level by 1. |
|
|
|
|
|
|
|
### Extracted Relations: |
|
{ |
|
"section_description": "The article discusses the various ways in which grass blocks can be used in Minecraft, including as food, fuel, and protection. It also covers how grass blocks can be used to tame animals, and the dangers of being exposed to grass. The article concludes by highlighting the importance of knowing how to use and defend against grass blocks.", |
|
"list_of_entities": [ |
|
"dirt", |
|
"Minecraft", |
|
"light level", |
|
"animal", |
|
"surface grass blocks", |
|
"Bedrock Edition", |
|
"Java Edition", |
|
"dirt path", |
|
"grass", |
|
"snow", |
|
"snowy grass block", |
|
"opaque blocks", |
|
"sheep", |
|
"composter", |
|
"transparent blocks" |
|
], |
|
"relations": [ |
|
{ |
|
"description": "Grass blocks are a renewable resource that can be used as a food item.", |
|
"source_entities": [ |
|
"Minecraft", |
|
"grass" |
|
], |
|
"target_entities": [ |
|
"animal", |
|
"surface grass blocks" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Grass blocks can be used to tame animals.", |
|
"source_entities": [ |
|
"Minecraft", |
|
"grass" |
|
], |
|
"target_entities": [ |
|
"animal", |
|
"surface grass blocks" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Exposure to grass blocks can cause damage to the player's health.", |
|
"source_entities": [ |
|
"Minecraft" |
|
], |
|
"target_entities": [ |
|
"grass", |
|
"player" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Placing grass blocks in a composter has a 30% chance of raising the compost level by 1.", |
|
"source_entities": [ |
|
"Minecraft", |
|
"grass" |
|
], |
|
"target_entities": [ |
|
"composter", |
|
"grass" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Surface grass blocks are the only ones that can be used to spawn animals.", |
|
"source_entities": [ |
|
"Minecraft" |
|
], |
|
"target_entities": [ |
|
"surface grass blocks", |
|
"animal" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Animals regularly spawn in most Overworld biomes on surface grass blocks that have light level 7 or brighter in the space directly above.", |
|
"source_entities": [ |
|
"Minecraft" |
|
], |
|
"target_entities": [ |
|
"surface grass blocks", |
|
"animal" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "If you walk into the grass without first getting rid of any animals or monsters that are there, they will attack you.", |
|
"source_entities": [ |
|
"Minecraft" |
|
], |
|
"target_entities": [ |
|
"grass", |
|
"player" |
|
], |
|
"strength": "moderate" |
|
}, |
|
{ |
|
"description": "Placing grass blocks with snow on top of them turns them into snowy grass blocks.", |
|
"source_entities": [ |
|
"Minecraft" |
|
], |
|
"target_entities": [ |
|
"snow", |
|
"grass" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Removing the snow layer turns the block into a normal grass block.", |
|
"source_entities": [ |
|
"Minecraft" |
|
], |
|
"target_entities": [ |
|
"snowy grass block", |
|
"grass" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Dirt path blocks can be created by using any type of shovel on the side or top of a grass block with air above it.", |
|
"source_entities": [ |
|
"Minecraft", |
|
"grass" |
|
], |
|
"target_entities": [ |
|
"dirt path", |
|
"shovel" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "The shovel loses 1 durability for each dirt path block created.", |
|
"source_entities": [ |
|
"Minecraft", |
|
"shovel" |
|
], |
|
"target_entities": [ |
|
"dirt path", |
|
"shovel" |
|
], |
|
"strength": "moderate" |
|
}, |
|
{ |
|
"description": "Death grass block dies and changes to dirt a random time (when a random tick lands on the block)", |
|
"source_entities": [ |
|
"Minecraft" |
|
], |
|
"target_entities": [ |
|
"death grass block", |
|
"dirt" |
|
], |
|
"strength": "strong" |
|
}, |
|
{ |
|
"description": "Grass can be used to create dirt paths", |
|
"source_entities": [ |
|
"Minecraft" |
|
], |
|
"target_entities": [ |
|
"grass", |
|
"dirt path" |
|
], |
|
"strength": "strong" |
|
} |
|
] |
|
} |
|
``` |
|
|
|
## Model Card Contact |
|
|
|
[More Information Needed] |
|
### Framework versions |
|
|
|
- PEFT 0.10.0 |
