Update README.md

README.md

@@ -1,198 +1,542 @@
 ---
 library_name: peft
 base_model: NousResearch/Llama-2-7b-hf
+license: llama2
+datasets:
+- DehydratedWater42/semantic_relations_extraction
+language:
+- en
+inference: false
+tags:
+- math
+- semantic
+- extraction
+- graph
+- relations
+- science
+- synthetic
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
+# SeELLama (Semantic Extraction LLama)
+Model based on LLama2-7b fine-tuned with `DehydratedWater42/semantic_relations_extraction` dataset.
+
+The purpose of this model is to extract semantic relations from text in structured way.
+
+
+### This is just adapter for `NousResearch/Llama-2-7b-hf`
+- **Get SeELLama as Safetensors:** [DehydratedWater42/SeELLama](https://huggingface.co/DehydratedWater42/SeELLama)
+- **Get SeELLama as GGUF:** [DehydratedWater42/SeELLama-GGUF](https://huggingface.co/DehydratedWater42/SeELLama-GGUF)
+
+## How to use it:
+### Template:
+```text
+Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format.
+### Text Part to Extract From:
+<<your_text_for_extraction>>
+### Extracted Relations:
+{
+  "section_description": 
+```
+The `JSON` opening is not necessary but it improves stability. Remember to use double `{{` instead of singular '{' if you are using LangChain prompts with fstring formatting.
+
+## Fine-tuning code/settings
+
+```python
+tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-hf", trust_remote_code=True)
+tokenizer.pad_token = tokenizer.eos_token
+tokenizer.padding_side = "right"
+
+torch.backends.cuda.matmul.allow_tf32 = True
+torch.backends.cudnn.allow_tf32 = True
+
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True, 
+    bnb_4bit_use_double_quant=True, 
+    bnb_4bit_quant_type="nf4", 
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    "NousResearch/Llama-2-7b-hf",
+    quantization_config=bnb_config,
+    use_cache=False,
+    use_flash_attention_2=False,
+    device_map="auto",
+)
+model.config.pretraining_tp = 1
+
+peft_params = LoraConfig(
+    lora_alpha=16,
+    lora_dropout=0.1,
+    r=64,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+
+training_params = TrainingArguments(
+    output_dir="./results",
+    num_train_epochs=3,
+    per_device_train_batch_size=1,
+    gradient_accumulation_steps=1,
+    optim="paged_adamw_32bit",
+    save_steps=100,
+    logging_steps=1,
+    learning_rate=2e-4,
+    weight_decay=0.001,
+    bf16=True,
+    fp16=False,
+    tf32=True,
+    max_grad_norm=0.3,
+    max_steps=-1,
+    warmup_ratio=0.03,
+    group_by_length=True,
+    lr_scheduler_type="constant",
+    report_to="mlflow",
+    run_name="semantic-extraction-llama2-7b"
+)
+
+trainer = SFTTrainer(
+    model=model,
+    train_dataset=formated_dataset,
+    peft_config=peft_params,
+    dataset_text_field="text",
+    max_seq_length=int(1024 * 2.5), # 2560
+    tokenizer=tokenizer,
+    args=training_params,
+    packing=True,
+)
+```
+
+### Example:
+Extracting information from Minecraft Wiki.
+
+#### Initial template
+```text
+Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format.
+### Text Part to Extract From:
+Players in Survival, Adventure, or Spectator have access to the 2×2 crafting grid from their inventory screen. Small crafting recipes that are at most 2×2 can be made there. These include wooden planks, sticks, crafting tables, torches and some shapeless recipes. To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it. This brings up an interface with a larger, 3×3 crafting grid, which the player can use to craft any crafting recipe in the game. There is also a recipe book where the player can access all the crafting recipes.
+
+Some recipes do not require their ingredients to be arranged in a specific way on the crafting grid. These are commonly known as shapeless recipes. For example, players may craft a fermented spider eye by placing its ingredients anywhere within the grid or like a flint and steel can be crafted in any orientation.
+
+On the contrary, many recipes must have their ingredients placed in the correct relative positions on the crafting grid. These are commonly known as shaped recipes. Ingredients in shaped recipes can be ‘moved’ up, down, left, or right. They can also be flipped side-ways. For example, a 3×1 recipe, such as bread, can be made using the top, middle, or bottom row of the 3×3 grid, and a bow may be made with the strings placed on the left instead of on the right.
+
+There are recipes that may not be moved or mirrored in this way. These are commonly known as fixed recipes. For example, dyes in banner recipes - only available in Bedrock Edition - must be specifically placed to achieve the desired pattern. Fixed recipes can be added by data packs, add-ons, or mods.
+### Extracted Relations:
+{
+  "section_description": 
+```
+
+
+#### Result (about game):
+```text
+Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format.
+### Text Part to Extract From:
+Players in Survival, Adventure, or Spectator have access to the 2×2 crafting grid from their inventory screen. Small crafting recipes that are at most 2×2 can be made there. These include wooden planks, sticks, crafting tables, torches and some shapeless recipes. To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it. This brings up an interface with a larger, 3×3 crafting grid, which the player can use to craft any crafting recipe in the game. There is also a recipe book where the player can access all the crafting recipes.
+
+Some recipes do not require their ingredients to be arranged in a specific way on the crafting grid. These are commonly known as shapeless recipes. For example, players may craft a fermented spider eye by placing its ingredients anywhere within the grid or like a flint and steel can be crafted in any orientation.
+
+On the contrary, many recipes must have their ingredients placed in the correct relative positions on the crafting grid. These are commonly known as shaped recipes. Ingredients in shaped recipes can be ‘moved’ up, down, left, or right. They can also be flipped side-ways. For example, a 3×1 recipe, such as bread, can be made using the top, middle, or bottom row of the 3×3 grid, and a bow may be made with the strings placed on the left instead of on the right.
+
+There are recipes that may not be moved or mirrored in this way. These are commonly known as fixed recipes. For example, dyes in banner recipes - only available in Bedrock Edition - must be specifically placed to achieve the desired pattern. Fixed recipes can be added by data packs, add-ons, or mods.
+### Extracted Relations:
+{
+  "section_description": "In Minecraft, players can craft items using the 2x2 crafting grid. There are some fixed recipes that require the player to place their ingredients in a specific position.",
+  "list_of_entities": [
+   "crafting recipes",
+   "player",
+   "fixed recipes",
+   "2x2 crafting grid",
+   "survival, adventure, or spectator players",
+   "wooden planks",
+   "torches",
+   "shaped recipes",
+   "small crafting recipes",
+   "shapeless recipes",
+   "wooden planks, sticks, crafting tables, torches",
+   "flint and steel"
+  ],
+  "relations": [
+   {
+     "description": "Players have access to the 2x2 crafting grid from their inventory screen.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "player"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Small crafting recipes that are at most 2x2 can be made using the 2x2 crafting grid.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "survival, adventure, or spectator players"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "wooden planks"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "crafting tables"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Crafting tables allow players to make bigger crafting recipes that are not possible with the 2x2 crafting grid.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "torches",
+       "flint and steel",
+       "wooden planks"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Crafting tables are used to craft items using the 2x2 crafting grid.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "wooden planks, sticks, crafting tables, torches"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "There are some fixed recipes that require the player to place their ingredients in a specific position.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "fixed recipes"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "The recipes do not require their ingredients to be arranged in a specific way on the crafting grid.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "shapeless recipes"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Ingredients in shapeless recipes can be ‘moved’ up, down, left, or right.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "shaped recipes"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Ingredients in shaped recipes can be flipped side-ways.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "fixed recipes"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "There are some fixed recipes that require the player to place their ingredients in a specific position.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "fixed recipes"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "The recipes can be added by data packs, add-ons, or mods.",
+     "source_entities": [
+       "crafting recipes"
+     ],
+     "target_entities": [
+       "fixed recipes"
+     ],
+     "strength": "strong"
+   }
+  ]
+}
+```
+
+### Smaller example (About crafting)
+```text
+Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format.
+### Text Part to Extract From:
+Players in Survival, Adventure, or Spectator have access to the 2×2 crafting grid from their inventory screen. Small crafting recipes that are at most 2×2 can be made there. These include wooden planks, sticks, crafting tables, torches and some shapeless recipes. To craft items using a bigger grid to allow more crafting recipes, create a crafting table with 4 wooden planks, place it in the world, and press use while facing it. This brings up an interface with a larger, 3×3 crafting grid, which the player can use to craft any crafting recipe in the game. There is also a recipe book where the player can access all the crafting recipes.
+### Extracted Relations:
+{
+  "section_description":  "Survival, Adventure, and Spectator players have access to the 2×2 crafting grid. However, to craft items using a bigger grid, you need to create a crafting table with wooden planks, place it in the world, and press use while facing it.",
+  "list_of_entities": [
+   "Survival",
+   "wooden planks",
+   "Spectator",
+   "Adventure",
+   "crafting grid"
+  ],
+  "relations": [
+   {
+     "description": "Survival players can access the 2×2 crafting grid.",
+     "source_entities": [
+       "Survival"
+     ],
+     "target_entities": [
+       "crafting grid"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Adventure and Spectator players can also access the 2×2 crafting grid.",
+     "source_entities": [
+       "Adventure"
+     ],
+     "target_entities": [
+       "crafting grid"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "To craft items using a bigger grid, you need to create a crafting table with wooden planks.",
+     "source_entities": [
+       "Spectator"
+     ],
+     "target_entities": [
+       "crafting grid"
+     ],
+     "strength": "strong"
+   }
+  ]
+}
+```
+
+### Longer example (dirt block)
+```text
+Below is an part of larger text. Your task is to extract information about entities and relations to the JSON format.
+### Text Part to Extract From:
+Usage
+Using bone meal on grass blocks causes short grass, tall grass,‌[Java Edition only] ferns,‌[Bedrock Edition only] and flowers to grow.
+
+Passive mobs tend to wander toward grass blocks. They also wander toward light.
+
+When a sheep eats a grass block, the block becomes dirt, and a sheared sheep regrows its wool. Baby sheep graze grass much more often than adults and mature 1 minute faster when grazing.
+
+Tilling a grass block with a hoe converts it to a farmland block.
+
+Dirt paths can be created by using any type of shovel on the side or top of a grass block with air above it. The shovel loses 1 durability for each dirt path block created.
+
+Death
+Grass dies and changes to dirt a random time (when a random tick lands on the block) after it has been covered by any opaque blocks. Transparent blocks can kill grass in a similar manner if they cause the light level above the grass block to be four or below (like water does), and the surrounding area is not otherwise sufficiently illuminated.
+
+Spawning
+In Java Edition, animals occasionally spawn on grass blocks that have light level 9 or brighter in the space directly above. This is quite rare and requires that there be few other animals nearby or in the spawn chunks. Most animals are instead created along with the terrain.
+
+In Bedrock Edition, most animals require surface grass blocks (grass blocks with no solid block anywhere above them at the same x, z coordinate) to spawn. Animals regularly spawn in most Overworld biomes on surface grass blocks that have light level 7 or brighter in the space directly above.
+
+Snowy grass block
+
+A snowy grass block with snow on top of it.
+Grass blocks turn into a snowy grass block if snow, powder snow or a snow block is placed on top of the block. Removing the snow layer turns the block into a normal grass block.
+
+Composting
+In Bedrock Edition, placing grass blocks into a composter has a 30% chance of raising the compost level by 1.
+
+
+
+### Extracted Relations:
+{
+  "section_description":  "The article discusses the various ways in which grass blocks can be used in Minecraft, including as food, fuel, and protection. It also covers how grass blocks can be used to tame animals, and the dangers of being exposed to grass. The article concludes by highlighting the importance of knowing how to use and defend against grass blocks.",
+  "list_of_entities": [
+   "dirt",
+   "Minecraft",
+   "light level",
+   "animal",
+   "surface grass blocks",
+   "Bedrock Edition",
+   "Java Edition",
+   "dirt path",
+   "grass",
+   "snow",
+   "snowy grass block",
+   "opaque blocks",
+   "sheep",
+   "composter",
+   "transparent blocks"
+  ],
+  "relations": [
+   {
+     "description": "Grass blocks are a renewable resource that can be used as a food item.",
+     "source_entities": [
+       "Minecraft",
+       "grass"
+     ],
+     "target_entities": [
+       "animal",
+       "surface grass blocks"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Grass blocks can be used to tame animals.",
+     "source_entities": [
+       "Minecraft",
+       "grass"
+     ],
+     "target_entities": [
+       "animal",
+       "surface grass blocks"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Exposure to grass blocks can cause damage to the player's health.",
+     "source_entities": [
+       "Minecraft"
+     ],
+     "target_entities": [
+       "grass",
+       "player"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Placing grass blocks in a composter has a 30% chance of raising the compost level by 1.",
+     "source_entities": [
+       "Minecraft",
+       "grass"
+     ],
+     "target_entities": [
+       "composter",
+       "grass"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Surface grass blocks are the only ones that can be used to spawn animals.",
+     "source_entities": [
+       "Minecraft"
+     ],
+     "target_entities": [
+       "surface grass blocks",
+       "animal"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Animals regularly spawn in most Overworld biomes on surface grass blocks that have light level 7 or brighter in the space directly above.",
+     "source_entities": [
+       "Minecraft"
+     ],
+     "target_entities": [
+       "surface grass blocks",
+       "animal"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "If you walk into the grass without first getting rid of any animals or monsters that are there, they will attack you.",
+     "source_entities": [
+       "Minecraft"
+     ],
+     "target_entities": [
+       "grass",
+       "player"
+     ],
+     "strength": "moderate"
+   },
+   {
+     "description": "Placing grass blocks with snow on top of them turns them into snowy grass blocks.",
+     "source_entities": [
+       "Minecraft"
+     ],
+     "target_entities": [
+       "snow",
+       "grass"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Removing the snow layer turns the block into a normal grass block.",
+     "source_entities": [
+       "Minecraft"
+     ],
+     "target_entities": [
+       "snowy grass block",
+       "grass"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Dirt path blocks can be created by using any type of shovel on the side or top of a grass block with air above it.",
+     "source_entities": [
+       "Minecraft",
+       "grass"
+     ],
+     "target_entities": [
+       "dirt path",
+       "shovel"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "The shovel loses 1 durability for each dirt path block created.",
+     "source_entities": [
+       "Minecraft",
+       "shovel"
+     ],
+     "target_entities": [
+       "dirt path",
+       "shovel"
+     ],
+     "strength": "moderate"
+   },
+   {
+     "description": "Death grass block dies and changes to dirt a random time (when a random tick lands on the block)",
+     "source_entities": [
+       "Minecraft"
+     ],
+     "target_entities": [
+       "death grass block",
+       "dirt"
+     ],
+     "strength": "strong"
+   },
+   {
+     "description": "Grass can be used to create dirt paths",
+     "source_entities": [
+       "Minecraft"
+     ],
+     "target_entities": [
+       "grass",
+       "dirt path"
+     ],
+     "strength": "strong"
+   }
+  ]
+}
+```
 
 ## Model Card Contact