base_model:
- mistralai/Mistral-Small-Instruct-2409
datasets:
- kejian/arxiv-physics-debug-v0
tags:
- unsloth
- physics
Mistral Physics Fine-tuned Model
This model is a fine-tuned version of mistralai/Mistral-Small-Instruct-2409 via Unsloth on kejian/arxiv-physics-debug-v0. Mostly for concept proofing, don't trust it for real physics (I mean, even Claude 3.5 can be wrong on graduate physics plenty of times, let alone a 22B model, but this hould perform a lot better than benhaotang/llama3.2-1B-physics-finetuned)!
Sorry for not having F16 version, there is no way to fit everything into VRAM or RAM at the same time in my current configuration.
Model description
- Base model: mistralai/Mistral-Small-Instruct-2409
- Training data: kejian/arxiv-physics-debug-v0
- Fine-tuning type: LoRA
- Use case: Physics domain questions
- Training Arguments:
- Learning Rate: 2e-5
- Epochs: 3
- Gradient Accumulation Steps: 8
- Training setup: Colab with Nvidia L4
Usage
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
load_in_8bit=False,
llm_int8_enable_fp32_cpu_offload=True
)
model = AutoModelForCausalLM.from_pretrained(
"benhaotang/mistral-small-physics-finetuned-bnb-4bit",
device_map="auto",
torch_dtype=torch.float16,
offload_folder="offload_folder",
quantization_config=bnb_config
)
tokenizer = AutoTokenizer.from_pretrained("benhaotang/mistral-small-physics-finetuned-bnb-4bit")
# Example usage
text = "Give me a short intodcution to renormalization group(RG) flow in physcis?\n"
inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=2048)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Example output:
The Renormalization Group (RG) flow is a fundamental concept in physics, particularly in the study of phase transitions and critical phenomena. It describes how the behavior of a system changes with scale, such as the size of the system or the length scale of fluctuations. Here's a brief introduction:
Scale Dependence: Many physical systems exhibit properties that depend on the scale at which they are observed. For example, a fluid might appear smooth on a large scale but exhibit turbulent behavior on a small scale.
RG Transformation: The RG flow involves a transformation that systematically integrates out short-distance fluctuations to obtain an effective theory that describes the system at longer distances. This is often done by progressively eliminating high-momentum modes in the system.
Fixed Points and Universality: The RG flow can lead to fixed points, which are scale-invariant solutions. Systems that flow to the same fixed point under RG transformation exhibit universal behavior, meaning their large-scale properties are the same regardless of the details of the system at small scales.
Relevant and Irrelevant Operators: In the vicinity of a fixed point, operators can be classified as relevant (grow under RG flow), irrelevant (shrink), or marginal (remain constant). Relevant operators drive the system away from the fixed point, while irrelevant ones become negligible at large scales.
Applications: RG flow is crucial in understanding critical phenomena, such as phase transitions in statistical mechanics, and has applications in condensed matter physics, quantum field theory, and even in areas like biology and computer science.
In essence, RG flow helps us understand how the microscopic details of a system influence its macroscopic behavior, and how universal properties emerge from complex systems.
Training
| Step | Training Loss | Validation Loss |
|---|---|---|
| 50 | 2.407400 | 1.798349 |
| 100 | 1.452000 | 1.765856 |
| 150 | 1.161300 | 1.716366 |
| 200 | 1.223700 | 1.704631 |
| 250 | 1.135900 | 1.683653 |
| 300 | 1.371900 | 1.677721 |
| 350 | 1.208500 | 1.657915 |
| 400 | 1.303400 | 1.657678 |
| 450 | 1.233700 | 1.642972 |
| 500 | 1.081900 | 1.653393 |
| 550 | 1.117700 | 1.645338 |
| 600 | 1.109500 | 1.651868 |
| 650 | 1.190100 | 1.689853 |
| 700 | 1.000000 | 1.663633 |
| 750 | 1.020100 | 1.647308 |
| 800 | 1.033400 | 1.675173 |
| 850 | 1.082300 | 1.652737 |
| 900 | 1.074000 | 1.665859 |
| 950 | 0.975300 | 1.661394 |
| 1000 | 0.955000 | 1.672116 |
| 1050 | 1.017000 | 1.656730 |
| 1100 | 0.941500 | 1.652197 |
| 1150 | 1.003100 | 1.657381 |
| 1200 | 0.891100 | 1.662021 |
| 1250 | 0.931000 | 1.662401 |
| 1300 | 0.932800 | 1.662421 |
| 1350 | 1.042000 | 1.665535 |