| # Uncased Finnish Sentence BERT model | |
| Finnish Sentence BERT trained from FinBERT | |
| ## Training | |
| FinBERT model: TurkuNLP/bert-base-finnish-uncased-v1 | |
| Data: The data provided [here] (https://turkunlp.org/paraphrase.html), including the Finnish Paraphrase Corpus and the automatically collected paraphrase candidates (500K positive and 5M negative) | |
| Pooling: mean pooling | |
| Task: Binary prediction, whether two sentences are paraphrases or not. Note: the labels 3 and 4 are considered paraphrases, and labels 1 and 2 non-paraphrases. [Details on labels] (https://aclanthology.org/2021.nodalida-main.29/) | |
| ## Usage | |
| The same as in [HuggingFace documentation] (https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens). Either through `SentenceTransformer` or `HuggingFace Transformers` | |
| ### SentenceTransformer | |
| ``` | |
| from sentence_transformers import SentenceTransformer | |
| sentences = ["Tämä on esimerkkilause.", "Tämä on toinen lause."] | |
| model = SentenceTransformer('TurkuNLP/sbert-uncased-finnish-paraphrase') | |
| embeddings = model.encode(sentences) | |
| print(embeddings) | |
| ``` | |
| ### HuggingFace Transformers | |
| ``` | |
| from transformers import AutoTokenizer, AutoModel | |
| import torch | |
| #Mean Pooling - Take attention mask into account for correct averaging | |
| def mean_pooling(model_output, attention_mask): | |
| token_embeddings = model_output[0] #First element of model_output contains all token embeddings | |
| input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float() | |
| return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9) | |
| # Sentences we want sentence embeddings for | |
| sentences = ["Tämä on esimerkkilause.", "Tämä on toinen lause."] | |
| # Load model from HuggingFace Hub | |
| tokenizer = AutoTokenizer.from_pretrained('TurkuNLP/sbert-uncased-finnish-paraphrase') | |
| model = AutoModel.from_pretrained('TurkuNLP/sbert-uncased-finnish-paraphrase') | |
| # Tokenize sentences | |
| encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') | |
| # Compute token embeddings | |
| with torch.no_grad(): | |
| model_output = model(**encoded_input) | |
| # Perform pooling. In this case, mean pooling. | |
| sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask']) | |
| print("Sentence embeddings:") | |
| print(sentence_embeddings) | |
| ``` | |