README.md

---
base_model: mrm8488/longformer-base-4096-finetuned-squadv2
tags:
  - generated_from_trainer
license: apache-2.0
datasets:
  - Kkordik/NovelQSI
language:
  - en
widget:
  - text: >-
      Which character said 'You know, I read somewhere that the brightest stars are those that have undergone the most turmoil. Maybe it's the same with us – our struggles make us shine brighter'?
    context: >-
      Characters:
      
      Alex: Aliases: {'Alex'}. Gender: Male, The character is: major
      Bella: Aliases: {'Bella'}. Gender: Female, The character is: major
      Charlie: Aliases: {'Charlie'}. Gender: Non-binary, The character is: major
      
      Summary: 
      
      In the novel's previous section, Alex, Bella, and Charlie, three friends in Luminara, engage in a deep conversation at The Starry Night café. They debate destiny, with Alex believing in fate, Bella advocating for self-made destiny, and Charlie suggesting a combination of both. Personal reflections emerge, such as Bella's musings on heartbreak and Alex's thoughts on longing. Charlie compares people's struggles to stars, implying that challenges enhance personal growth. The night progresses with their varied, meaningful discussions.
      
      Novel Text:
      
      In the bustling city of Luminara, three friends, Alex, Bella, and Charlie, often met at their favorite café, The Starry Night, to discuss life, love, and the mysteries of the universe. The café, with its warm ambiance and the soft hum of jazz in the background, provided the perfect setting for their deep conversations.
      
      One evening, as the city lights twinkled outside, the trio found themselves engrossed in a discussion about destiny.
      
      Alex, a firm believer in fate, argued passionately, "I truly believe that our paths are predestined. The universe has a plan for each of us, and all our choices lead us to our ultimate destiny."
      
      Bella, a skeptic, laughed softly and countered, "That's a romantic notion, Alex, but I think we make our own destiny. It's our decisions, not some cosmic plan, that shape our lives."
      
      Charlie, always the mediator, added thoughtfully, "Maybe it's a bit of both. Perhaps there's a grand design, but within it, we have the freedom to make choices that influence our journey."
      
      Their conversation drifted to other topics as the evening wore on. At one point, Bella, reflecting on a recent heartbreak, said, "Sometimes, I wonder if the heart ever truly heals from loss, or if it just learns to live with the pain."
      
      Alex, looking out the window at the starry sky, mused, "It's strange how the heart yearns for what it can't have. The unattainable always seems so much more alluring."
      
      Charlie, who had been quiet for a while, suddenly spoke up with a gleam in his eye, "You know, I read somewhere that the brightest stars are those that have undergone the most turmoil. Maybe it's the same with us – our struggles make us shine brighter."
      
      As the night deepened, their conversation meandered through various topics.
model-index:
  - name: Kkordik/test_longformer_4096_qsi
    results:
      - task:
          type: question-answering
        dataset:
          type: Kkordik/NovelQSI
          name: NovelQSI
          split: test
        metrics:
          - type: exact_match
            value: 20.346
            verified: false
          - type: f1
            value: 26.58
            verified: false

---

# longformer_4096_qsi

This model is a fine-tuned version of [mrm8488/longformer-base-4096-finetuned-squadv2](https://huggingface.co/mrm8488/longformer-base-4096-finetuned-squadv2) on a tiny [NovelQSI](https://huggingface.co/datasets/Kkordik/NovelQSI) dataset.
It achieves the following results on the evaluation set:
- Loss: 2.9598

## Model description

This model is a test model for my research project. The idea of the model is to understand which novel character said the requested quote. 
It achieves a bit better results on the ´test´ split of the NovelQSI dataset than base longformer-base-4096-finetuned-squadv2 model on the same dataset split.

**Base model results:**

```
{
  "exact_match": {
    "confidence_interval": [8.754452551305853, 14.718614718614718],
    "score": 12.121212121212121,
    "standard_error": 1.8579217243778676
  },
  "f1": {
    "confidence_interval": [18.469101076147584, 28.28409063313956],
    "score": 22.799422799422796,
    "standard_error": 2.896728175757627
  },
  "latency_in_seconds": 0.7730605573419919,
  "samples_per_second": 1.2935597224598967,
  "total_time_in_seconds": 178.5769887460001
}

```

 **Achieved results:**
 
```
{
  "exact_match": {
    "confidence_interval": [16.017316017316016, 24.242424242424242],
    "score": 20.346320346320347,
    "standard_error": 2.9434375492784994
  },
  "f1": {
    "confidence_interval": [23.123469058324783, 31.823648733317036],
    "score": 26.580086580086572,
    "standard_error": 2.593030474995015
  },
  "latency_in_seconds": 0.8093855569913422,
  "samples_per_second": 1.235505120349827,
  "total_time_in_seconds": 186.96806366500005
}
```

The results have shown, that the technique has its future.

## Training and evaluation data

You can find training code in the github repo of my research:

https://github.com/Kkordik/NovelQSI

It was trained and evaluated in notebooks, so it is easy to reproduce.

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 93   | 3.0886          |
| No log        | 1.99  | 186  | 3.3755          |
| No log        | 2.99  | 279  | 2.9598          |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu118
- Datasets 2.15.0
- Tokenizers 0.15.0