|
--- |
|
tags: salesken |
|
license: apache-2.0 |
|
inference: false |
|
|
|
--- |
|
|
|
We have trained a model to evaluate if a paraphrase is a semantic variation to the input query or just a surface level variation. Data augmentation by adding Surface level variations does not add much value to the NLP model training. if the approach to paraphrase generation is "OverGenerate and Rank" , Its important to have a robust model of scoring/ ranking paraphrases. NLG Metrics like bleu ,BleuRT, gleu , Meteor have not proved very effective in scoring paraphrases. |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
import torch |
|
import pandas as pd |
|
import numpy as np |
|
tokenizer = AutoTokenizer.from_pretrained("salesken/paraphrase_diversity_ranker") |
|
model = AutoModelForSequenceClassification.from_pretrained("salesken/paraphrase_diversity_ranker") |
|
|
|
input_query = ["tough challenges make you stronger."] |
|
paraphrases = [ |
|
"tough problems make you stronger", |
|
"tough problems will make you stronger", |
|
"tough challenges make you stronger", |
|
"tough challenges will make you a stronger person", |
|
"tough challenges will make you stronger", |
|
"tough tasks make you stronger", |
|
"the tough task makes you stronger", |
|
"tough stuff makes you stronger", |
|
"if tough times make you stronger", |
|
"the tough part makes you stronger", |
|
"tough issues strengthens you", |
|
"tough shit makes you stronger", |
|
"tough tasks force you to be stronger", |
|
"tough challenge is making you stronger", |
|
"tough problems make you have more strength"] |
|
para_pairs=list(pd.MultiIndex.from_product([input_query, paraphrases])) |
|
|
|
|
|
features = tokenizer(para_pairs, padding=True, truncation=True, return_tensors="pt") |
|
model.eval() |
|
with torch.no_grad(): |
|
scores = model(**features).logits |
|
label_mapping = ['surface_level_variation', 'semantic_variation'] |
|
labels = [label_mapping[score_max] for score_max in scores.argmax(dim=1)] |
|
|
|
sorted_diverse_paraphrases= np.array(para_pairs)[scores[:,1].sort(descending=True).indices].tolist() |
|
print(sorted_diverse_paraphrases) |
|
|
|
|
|
# to identify the type of paraphrase (surface-level variation or semantic variation) |
|
print("Paraphrase type detection=====", list(zip(para_pairs, labels))) |
|
``` |
|
|
|
============================================================================ |
|
|
|
For more robust results, filter out the paraphrases which are not semantically |
|
similar using a model trained on NLI, STS task and then apply the ranker . |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelWithLMHead |
|
from transformers import AutoModelForSequenceClassification |
|
from sentence_transformers import SentenceTransformer, util |
|
import torch |
|
import pandas as pd |
|
import numpy as np |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("salesken/paraphrase_diversity_ranker") |
|
model = AutoModelForSequenceClassification.from_pretrained("salesken/paraphrase_diversity_ranker") |
|
embedder = SentenceTransformer('stsb-bert-large') |
|
|
|
input_query = ["tough challenges make you stronger."] |
|
paraphrases = [ |
|
"tough problems make you stronger", |
|
"tough problems will make you stronger", |
|
"tough challenges make you stronger", |
|
"tough challenges will make you a stronger person", |
|
"tough challenges will make you stronger", |
|
"tough tasks make you stronger", |
|
"the tough task makes you stronger", |
|
"tough stuff makes you stronger", |
|
"tough people make you stronger", |
|
"if tough times make you stronger", |
|
"the tough part makes you stronger", |
|
"tough issues strengthens you", |
|
"tough shit makes you stronger", |
|
"tough tasks force you to be stronger", |
|
"tough challenge is making you stronger", |
|
"tough problems make you have more strength"] |
|
|
|
|
|
corpus_embeddings = embedder.encode(paraphrases, convert_to_tensor=True) |
|
query_embedding = embedder.encode(input_query, convert_to_tensor=True) |
|
cos_scores = util.pytorch_cos_sim(query_embedding, corpus_embeddings)[0] |
|
para_set=np.array(paraphrases) |
|
a=cos_scores.sort(descending=True) |
|
para= para_set[a.indices[a.values>=0.7].cpu()].tolist() |
|
|
|
|
|
para_pairs=list(pd.MultiIndex.from_product([input_query, para])) |
|
|
|
import torch |
|
features = tokenizer(para_pairs, padding=True, truncation=True, return_tensors="pt") |
|
model.eval() |
|
with torch.no_grad(): |
|
scores = model(**features).logits |
|
label_mapping = ['surface_level_variation', 'semantic_variation'] |
|
labels = [label_mapping[score_max] for score_max in scores.argmax(dim=1)] |
|
|
|
sorted_diverse_paraphrases= np.array(para)[scores[:,1].sort(descending=True).indices].tolist() |
|
print("Paraphrases sorted by diversity:=======",sorted_diverse_paraphrases) |
|
|
|
# to identify the type of paraphrase (surface-level variation or semantic variation) |
|
print("Paraphrase type detection=====", list(zip(para_pairs, labels))) |
|
|
|
``` |
