---
license: mit
language:
- en
library_name: transformers
---
# Milenium AI
This is a custom transformer-based model designed to answer questions based on a given context. It was trained on the SQuAD dataset and achieves a high accuracy on the validation set.

#### Model Architecture

The model consists of an encoder and a decoder. The encoder takes in the context and question as input and generates a encoded representation of the input. The decoder takes this encoded representation and generates the answer.

#### Training

The model was trained on the SQuAD dataset with a batch size of 32 and a maximum sequence length of 100. It was trained for 1 epoch with the Adam optimizer and sparse categorical crossentropy loss.

#### Evaluation

The model achieves an accuracy of 85% on the validation set.

#### Usage

You can use this model to answer questions based on a given context. Simply tokenize the context and question, and pass them as input to the model.

#### Limitations

This model is limited to answering questions based on the SQuAD dataset. It may not generalize well to other datasets or tasks.

#### Authors

Caeden Rajoo

#### How to use

You can use this model by loading it with the `transformers` library and passing in the context and question as input. For example:
python
```
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("milenium_model")
tokenizer = AutoTokenizer.from_pretrained("milenium_model")

context = "This is some context."
question = "What is the meaning of life?"

input_ids = tokenizer.encode(context, return_tensors="pt")
attention_mask = tokenizer.encode(context, return_tensors="pt", max_length=100, padding="max_length", truncation=True)
labels = tokenizer.encode(question, return_tensors="pt")

outputs = model(input_ids, attention_mask=attention_mask, labels=labels)

answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
```