Upload 10 files

README.md

@@ -1,186 +1,94 @@
 ---
-language:
-- en
+pipeline_tag: sentence-similarity
 tags:
-- music
----
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-Embeds song lyrics to 300 dimensions.
-
-# Model Details
-
-## Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-- **Developed by:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** bert-base-uncased trained with contrastive learning
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-## Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-# Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-## Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-## Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-## Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-# Bias, Risks, and Limitations
-
-## Translate to English:
-chlussendlich existieren die Lyrics für 606'255 Songs. Um das weitere Vorgehen zu vereinfachen, wurden diese Songs durch die Python-Implementierung eines in Java implementierten Google Sprachdetektors \cite{nakatani2010langdetect} \cite{langdetectpy} gefiltert und nur die verbleibenden 480'964 englischen Lyrics werden weiter beachtet. 
-
-\subsection{Weitere Probleme}
-
-Im Nachhinein wurden 109 Lyrics festgestellt, die Spezialcharaktere haben, welche nicht vom Cleanup fetgestellt wurden. Diese wurden mit dem Regex \glqq '[a-zA-Z|\'|0-9]'\grqq{} gematcht und im Training ignoriert. Im Training wurden aber trotzdem einige Lyrics miteinberechnet, die zwar keine Spezialcharaktere haben, aber nicht ganz Englisch sind. Dadurch encoded das Languagemodel auch Japanische / Koreanische / Chinesische / Russische / Griechische sowie Spezialcharakter aus lateinischer Sprachen, jedoch mit sehr wenigen Trainingsdaten. Diese Lyrics wurden nicht durch das Google Spracherkennungsmodell als \glqq nicht Englisch\grqq{} eingestuft, weil sie genügend englische Wörter haben. Wir nehmen an, dass diese Lyrics das Training nicht gross beeinflussen und man kann von circa 500 solcher Songs ausgehen.
-
-Einige Lyrics sind auch lateinigiserte Versionen von japanischen / koreanischen / chinesischen Lieder (manuell geprüft). Weitere Grenzfälle sind Lyrics mit akzentuierten Lyrics wie:
-\\[8pt]
-\glqq let your fists swang k i c k y o a s s oh yes k i c k y o a s s oh yes i say beat you say that ass\grqq{}
-\\[8pt]
-Eine Analyse fehlt über was genau mit diesen Wörtern im Embedding Space passiert.
-
-## Recommendations
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- transformers
 
-bias, risk, technical limitations...
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-# Training Details
-
-## Training Data
-
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-
-
-[More Information Needed]
-
-## Training Procedure [optional]
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-### Preprocessing
-
-[More Information Needed]
-
-### Speeds, Sizes, Times
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-# Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-## Testing Data, Factors & Metrics
-
-### Testing Data
-
-<!-- This should link to a Data Card if possible. -->
-
-[More Information Needed]
-
-### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+---
 
-### Metrics
+# {MODEL_NAME}
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 300 dimensional dense vector space and can be used for tasks like clustering or semantic search.
 
-[More Information Needed]
+<!--- Describe your model here -->
 
-## Results
+## Usage (Sentence-Transformers)
 
-[More Information Needed]
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
 
-### Summary
+```
+pip install -U sentence-transformers
+```
 
+Then you can use the model like this:
 
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["This is an example sentence", "Each sentence is converted"]
 
-# Model Examination [optional]
+model = SentenceTransformer('{MODEL_NAME}')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
 
-<!-- Relevant interpretability work for the model goes here -->
 
-[More Information Needed]
 
-# Environmental Impact
+## Usage (HuggingFace Transformers)
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+#Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
 
-# Technical Specifications [optional]
 
-## Model Architecture and Objective
+# Sentences we want sentence embeddings for
+sentences = ['This is an example sentence', 'Each sentence is converted']
 
-[More Information Needed]
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
+model = AutoModel.from_pretrained('{MODEL_NAME}')
 
-## Compute Infrastructure
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
 
-[More Information Needed]
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
 
-### Hardware
+# Perform pooling. In this case, mean pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
 
-[More Information Needed]
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
 
-### Software
 
-[More Information Needed]
 
-# Citation [optional]
+## Evaluation Results
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+<!--- Describe how your model was evaluated -->
 
-**BibTeX:**
+For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
 
-[More Information Needed]
 
-# Model Card Contact
 
-for more info contact
-brunokreiner@hotmail.ch
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 300, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+)
+```
 
+## Citing & Authors
 
+<!--- Describe where people can find more information -->

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "/notebooks/p5/lyrics-bert-new-90000/90000/0_SentenceTransformer/",
+  "add_cross_attention": true,
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 300,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": true,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 2,
+  "num_hidden_layers": 2,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.20.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 52000
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.2",
+    "transformers": "4.20.1",
+    "pytorch": "1.12.0+cu116"
+  }
+}

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9a2878052f3261475f932eccc2e45fd9159dd3b74e60d69bf5e8d6330b7a23c
+size 83968433

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "name_or_path": "/notebooks/p5/lyrics-bert-new-90000/90000/0_SentenceTransformer/",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "special_tokens_map_file": null,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}