KhalidKhader/GPU-Project-phi-2-BZU-optimized-inference_opt-optimized

Optimized KhalidKhader/GPU-Project-phi-2-BZU-optimized

This is an optimized version of the original model using inference_opt optimization.

Performance

• Average inference time: 954.14ms
• Optimization method: inference_opt
• GPU: Tesla T4
• Precision: FP16

Quick Start

# Run the inference script
exec(open("inference.py").read())

# Generate text
result = generate_text("Your prompt here")
print(result)

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model
tokenizer = AutoTokenizer.from_pretrained("./")
model = AutoModelForCausalLM.from_pretrained("./", torch_dtype=torch.float16).to("cuda")
model.eval()

# Apply optimization
# Basic optimizations applied

# Generate text
def generate_text(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    with torch.no_grad():
        outputs = model.generate(inputs['input_ids'], max_new_tokens=50)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

result = generate_text("The future of AI is")
print(result)

Files

• pytorch_model.bin - Optimized model weights
• config.json - Model configuration
• tokenizer.json - Tokenizer
• inference.py - Ready-to-use inference script
• README.md - This file

Requirements

• PyTorch >= 2.0
• Transformers
• CUDA GPU