Transferred from Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -0,0 +1,129 @@
+---
+base_model: aubmindlab/bert-base-arabertv02
+datasets:
+- akhooli/arabic-triplets-1m-curated-sims-len
+language:
+- ar
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- transformers.js
+- transformers
+- sentence-similarity
+- feature-extraction
+- dataset_size:75000
+- loss:MatryoshkaLoss
+- loss:MultipleNegativesRankingLoss
+- mteb
+license: apache-2.0
+---
+
+# Arabic Triplet Matryoshka V2 Model [ATM2]
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/628f7a71dd993507cfcbe587/FrLQzFUJ3grEUOdONWGME.png)
+
+## Model Description
+
+Arabic-Triplet-Matryoshka-V2-Model is a state-of-the-art Arabic language embedding model based on the [sentence-transformers](https://www.SBERT.net) framework. It is fine-tuned from [aubmindlab/bert-base-arabertv02](https://huggingface.co/aubmindlab/bert-base-arabertv02) and specifically designed to capture the rich semantic nuances of Arabic text.
+
+This model maps sentences and paragraphs to a 768-dimensional dense vector space, enabling high-quality semantic text operations including:
+- Semantic textual similarity
+- Semantic search
+- Paraphrase mining
+- Text classification
+- Clustering
+- Information retrieval
+- Question answering
+
+## Key Features
+
+- **State-of-the-Art Performance**: Achieved 0.85 on STS17 and 0.64 on STS22.v2 with an average score of 74.5, making it the leading Arabic embedding model currently available.
+- **MatryoshkaLoss Training**: Utilizes nested embedding learning techniques to create hierarchical embeddings at multiple resolutions.
+- **Optimization**: Trained for 3 epochs with a final training loss of 0.718.
+- **Full Arabic Language Support**: Designed specifically to handle the complexity and morphological richness of Arabic language.
+
+## Training Details
+
+The model was trained using a combination of two loss functions:
+- **MatryoshkaLoss**: Enables the creation of nested embeddings at multiple resolutions, allowing for efficient and adaptable representations.
+- **MultipleNegativesRankingLoss**: Enhances the model's ability to discriminate between semantically similar and dissimilar text pairs.
+
+Training parameters:
+- **Base model**: aubmindlab/bert-base-arabertv02
+- **Dataset**: akhooli/arabic-triplets-1m-curated-sims-len (1M samples)
+- **Epochs**: 3
+- **Final Loss**: 0.718
+- **Embedding Dimension**: 768
+
+## Performance
+
+The model demonstrates exceptional performance on standard Arabic semantic textual similarity benchmarks:
+- **STS17**: 0.85
+- **STS22.v2**: 0.64
+- **Average Performance**: 74.5
+
+This represents the current state-of-the-art for Arabic embedding models, outperforming previous approaches by a significant margin.
+
+## Use Cases
+
+This model is particularly well-suited for:
+- **Information Retrieval**: Enhancing search capabilities for Arabic content.
+- **Document Similarity**: Identifying similar documents or text passages.
+- **Text Classification**: Powering classification systems for Arabic content.
+- **Question Answering**: Supporting Arabic QA systems with improved semantic understanding.
+- **Semantic Clustering**: Organizing Arabic text data based on meaning.
+- **Cross-lingual Applications**: When combined with other language models for multilingual applications.
+
+## Usage Examples
+
+```python
+from sentence_transformers import SentenceTransformer
+
+# Download from the 🤗 Hub
+model = SentenceTransformer("Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2")
+# Run inference
+sentences = [
+    'SENTENCE 1',
+    'SENTENCE 2',
+    'SENTENCE 3',
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 768]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities.shape)
+# [3, 3]
+```
+
+## Limitations
+
+Despite its strong performance, users should be aware of the following limitations:
+- The model may not perform optimally on highly technical or domain-specific Arabic text that was underrepresented in the training data.
+- As with all embedding models, performance may vary across different Arabic dialects and regional variations.
+- The model is optimized for semantic similarity tasks and may require fine-tuning for other specific applications.
+
+## Ethical Considerations
+
+This model is intended for research and applications that benefit Arabic language processing. Users should be mindful of potential biases that may exist in the training data and the resulting embeddings. We encourage responsible use of this technology and welcome feedback on ways to improve fairness and representation.
+
+## Citation
+
+If you use the Arabic Matryoshka Embeddings Model in your research or applications, please cite it as follows:
+
+```bibtex
+@article{nacar2024enhancing,
+  title={Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning},
+  author={Nacar, Omer and Koubaa, Anis},
+  journal={arXiv preprint arXiv:2407.21139},
+  year={2024}
+}
+```
+
+## Acknowledgements
+
+We would like to acknowledge [AraBERT](https://github.com/aub-mind/arabert) for the base model and [akhooli](https://huggingface.co/akhooli) for the valuable dataset that made this work possible.
+

config.json

@@ -0,0 +1,25 @@
+{
+  "_name_or_path": "aubmindlab/bert-base-arabertv02",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.43.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 64000
+}

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "3.0.1",
+    "transformers": "4.43.1",
+    "pytorch": "2.2.2"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": null
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee761022a6b8fc559f75ccb1ebb143e695acdf7d2263e41b6ba1535c9c131798
+size 540795752

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,86 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "[رابط]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": true,
+      "special": true
+    },
+    "6": {
+      "content": "[بريد]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": true,
+      "special": true
+    },
+    "7": {
+      "content": "[مستخدم]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": true,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "mask_token": "[MASK]",
+  "max_len": 512,
+  "model_max_length": 512,
+  "never_split": [
+    "[بريد]",
+    "[مستخدم]",
+    "[رابط]"
+  ],
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}