First model version

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -0,0 +1,168 @@
+---
+pipeline_tag: sentence-similarity
+language: "multilingual"
+tags:
+- feature-extraction
+- sentence-similarity
+- transformers
+- multilingual
+---
+
+# mstsb-paraphrase-multilingual-mpnet-base-v2
+
+This is a fine-tuned version of `paraphrase-multilingual-mpnet-base-v2` from [sentence-transformers](https://www.SBERT.net) model with [Semantic Textual Similarity Benchmark](http://ixa2.si.ehu.eus/stswiki/index.php/Main_Page) extended to 15 languages: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering, semantic search and measuring the similarity between two sentences.
+
+<!--- Describe your model here -->
+This model is fine-tuned version of `paraphrase-multilingual-mpnet-base-v2` for semantic textual similarity with multilingual data. The dataset used for this fine-tuning is STSb extended to 15 languages with Google Translator. For mantaining data quality the sentence pairs with a confidence value below 0.7 were dropped. The extended dataset is available at [GitHub](https://github.com/Huertas97/Multilingual-STSB). The languages included in the extended version are: ar, cs, de, en, es, fr, hi, it, ja, nl, pl, pt, ru, tr, zh-CN, zh-TW. The pooling operation used to condense the word embeddings into a sentence embedding is mean pooling (more info below). 
+
+<!-- ## Usage (Sentence-Transformers)
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+
+```python
+from sentence_transformers import SentenceTransformer
+# It support several languages
+sentences = ["This is an example sentence", "Esta es otra frase de ejemplo", "最後の例文"]
+
+# The pooling technique is automatically detected (mean pooling)
+model = SentenceTransformer('mstsb-paraphrase-multilingual-mpnet-base-v2')
+embeddings = model.encode(sentences)
+print(embeddings)
+``` -->
+
+
+
+## Usage (HuggingFace Transformers)
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+# We should define the proper pooling function: Mean pooling
+# Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+
+# Sentences we want sentence embeddings for
+sentences = ["This is an example sentence", "Esta es otra frase de ejemplo", "最後の例文"]
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('AIDA-UPM/mstsb-paraphrase-multilingual-mpnet-base-v2')
+model = AutoModel.from_pretrained('AIDA-UPM/mstsb-paraphrase-multilingual-mpnet-base-v2')
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling. In this case, max pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
+
+
+
+## Evaluation Results
+
+<!--- Describe how your model was evaluated -->
+Check the test results in the Semantic Textual Similarity Tasks. The 15 languages available at the [Multilingual STSB](https://github.com/Huertas97/Multilingual-STSB) have been combined into monolingual and cross-lingual tasks, giving a total of 31 tasks. Monolingual tasks have both sentences from the same language source (e.g., Ar-Ar, Es-Es), while cross-lingual tasks have two sentences, each in a different language being one of them English (e.g., en-ar, en-es). For the sake of readability tasks have been splitted into monolingual and cross-lingual tasks. 
+
+| Monolingual Task | Pearson Cosine test | Spearman Cosine  test |
+|------------------|---------------------|-----------------------|
+| en;en            | 0.868048310692506   | 0.8740170943535747    |
+| ar;ar            | 0.8267139454193487  | 0.8284459741532022    |
+| cs;cs            | 0.8466821720942157  | 0.8485417688803879    |
+| de;de            | 0.8517285961812183  | 0.8557680051557893    |
+| es;es            | 0.8519185309064691  | 0.8552243211580456    |
+| fr;fr            | 0.8430951067985064  | 0.8466614534379704    |
+| hi;hi            | 0.8178258630578092  | 0.8176462079184331    |
+| it;it            | 0.8475909574305637  | 0.8494216064459076    |
+| ja;ja            | 0.8435588859386477  | 0.8456031494178619    |
+| nl;nl            | 0.8486765104527032  | 0.8520856765262531    |
+| pl;pl            | 0.8407840177883407  | 0.8443070467300299    |
+| pt;pt            | 0.8534880178249296  | 0.8578544068829622    |
+| ru;ru            | 0.8390897585455678  | 0.8423041443534423    |
+| tr;tr            | 0.8382125451820572  | 0.8421587450058385    |
+| zh-CN;zh-CN      | 0.826233678946644   | 0.8248515460782744    |
+| zh-TW;zh-TW      | 0.8242683809675422  | 0.8235506799952028    |
+
+\\
+
+| Cross-lingual Task | Pearson Cosine test | Spearman Cosine  test |
+|--------------------|---------------------|-----------------------|
+| en;ar              | 0.7990830340462535  | 0.7956792016468148    |
+| en;cs              | 0.8381274879061265  | 0.8388713450024455    |
+| en;de              | 0.8414439600928739  | 0.8441971698649943    |
+| en;es              | 0.8442337511356952  | 0.8445035292903559    |
+| en;fr              | 0.8378437644605063  | 0.8387903367907733    |
+| en;hi              | 0.7951955086055527  | 0.7905052217683244    |
+| en;it              | 0.8415686372978766  | 0.8419480899107785    |
+| en;ja              | 0.8094306665283388  | 0.8032512280936449    |
+| en;nl              | 0.8389526140129767  | 0.8409310421803277    |
+| en;pl              | 0.8261309163979578  | 0.825976253023656     |
+| en;pt              | 0.8475546209070765  | 0.8506606391790897    |
+| en;ru              | 0.8248514914263723  | 0.8224871183202255    |
+| en;tr              | 0.8191803661207868  | 0.8194200775744044    |
+| en;zh-CN           | 0.8147678083378249  | 0.8102089470690433    |
+| en;zh-TW           | 0.8107272160374955  | 0.8056129680510944    |
+
+
+## Training
+The model was trained with the parameters:
+
+**DataLoader**:
+
+`torch.utils.data.dataloader.DataLoader` of length 687 with parameters:
+```
+{'batch_size': 132, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
+```
+
+**Loss**:
+
+`sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss` 
+
+Parameters of the fit()-Method:
+```
+{
+    "callback": null,
+    "epochs": 2,
+    "evaluation_steps": 1000,
+    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
+    "max_grad_norm": 1,
+    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
+    "optimizer_params": {
+        "lr": 2e-05
+    },
+    "scheduler": "WarmupLinear",
+    "steps_per_epoch": null,
+    "warmup_steps": 140,
+    "weight_decay": 0.01
+}
+```
+
+
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+)
+```
+
+## Citing & Authors
+
+<!--- Describe where people can find more information -->

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "/home/alvaro/.cache/torch/sentence_transformers/sentence-transformers_paraphrase-multilingual-mpnet-base-v2/",
+  "architectures": [
+    "XLMRobertaModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.8.2",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.0.0",
+    "transformers": "4.7.0",
+    "pytorch": "1.9.0+cu102"
+  }
+}

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:990cef9b421d6acfca4f685edfb089bb527b269c44dd9c3f03ce549f3bcf9215
+size 1112259155

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 128,
+  "do_lower_case": false
+}

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": false}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "sep_token": "</s>", "cls_token": "<s>", "unk_token": "<unk>", "pad_token": "<pad>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "/home/alvaro/.cache/torch/sentence_transformers/sentence-transformers_paraphrase-multilingual-mpnet-base-v2/", "tokenizer_class": "XLMRobertaTokenizer"}