FINGU-AI/FinguEm7b

SentenceTransformer based on Alibaba-NLP/gte-Qwen2-7B-instruct

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-Qwen2-7B-instruct. It maps sentences & paragraphs to a 3584-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Alibaba-NLP/gte-Qwen2-7B-instruct
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 3584 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: PeftModelForFeatureExtraction 
  (1): Pooling({'word_embedding_dimension': 3584, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '["Question: Find the remainder when $r^{13} + 1$ is divided by $r - 1$.\\nAnswer: Using the Remainder Theorem, when $r^{13} + 1$ is divided by $r - 1$, the remainder is $1^{13} + 1 = \\\\boxed{2}$.\\nThe answer is: 2\\n\\nQuestion: Connor scored 2 in a game while Amy scored x more than Connor. Jason scored twice the score of Amy. If Connor, Amy, and Jason belong to the same team, how many points did their team have?\\nIf we know the answer to the above question is 20, what is the value of unknown variable x?\\nAnswer: Connor scored 2 points in the game.\\nAmy scored x more than Connor, so her score is 2 + x.\\nJason scored twice the score of Amy, so his score is 2 * (2 + x) = 4 + 2x.\\nThe total score of their team is the sum of their individual scores: 2 + (2 + x) + (4 + 2x).\\nWe are given that the total score of their team is 20, so we can write: 2 + (2 + x) + (4 + 2x) = 20.\\nSimplifying the left side, we get: 8 + 3x = 20.\\nSubtracting 8 from both sides, we get: 3x = 12.\\nDividing both sides by 3, we get: x = 4.\\nThe value of x is 4.\\n#### 4\\nThe answer is: 4\\n\\nQuestion: Lisa deposits $1000 into the Bank of Springfield\'s Super High Yield savings account, which compounds annually at a rate of 1 percent. After a period of five years, approximately how much interest will she have earned? (Round your answer to the nearest dollar.)\\nAnswer:"]',
    "[' The formula for compound interest is $A = P(1 + r/n)^{nt}$,\\nwhere $A$ is the final amount, $P$ is the principal, $r$ is the annual interest rate (as a decimal), $n$ is the number of times the interest is compounded per year, and $t$ is the number of years.\\nIn this case, $P = 1000$, $r = 0.01$, $n = 1$, and $t = 5$.\\nPlugging in these values, we have $A = 1000(1 + 0.01/1)^{1*5}$.\\nEvaluating, we get $A \\\\approx 1000(1.01)^5 \\\\approx 1000(1.05101005) \\\\approx 1051.01$.\\nThe interest earned is $A - P = 1051.01 - 1000 = 51.01$.\\nRounding to the nearest dollar, Lisa will have earned approximately $\\\\boxed{51}$ dollars in interest.\\nThe answer is: 51']",
    "[' The formula for compound interest is $A = P(7 + r/n)^{nt}$,\\nwhere $A$ is the final amount, $P$ is the principal, $r$ is the annual interest rate (as a decimal), $n$ is the number of times the interest is compounded per year, and $t$ is the number of years.\\nIn this case, $P = 5593$, $r = 5.19$, $n = 6$, and $t = 1$.\\nPlugging in these values, we have $A = 3387(1 + 0.01/1)^{1*5}$.\\nEvaluating, we get $A \\\\approx 1000(1.01)^5 \\\\approx 1000(1.05101005) \\\\approx 1051.01$.\\nThe interest earned is $A - P = 1416.27 - 1000 = 88.88$.\\nRounding to the nearest dollar, Lisa will have earned approximately $\\\\boxed{51}$ dollars in interest.\\nThe answer is: 51']",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 3584]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}