Persian - Sentence Transformers
Creators: name: Farahani, Mehrdad
Publisher: name: Zenodo
Year: 2020
Subjects: BERT
ParsBERT
Sentence-Transformers
Wiki-Triplets
Persian Wikipedia
Semantic Similarity
Topic Modeling

description: <strong>Persian - Sentence Transformers </strong> There are many achievements in the Natural Language field for English or even other languages, but in the case of Persian, as you can see, there are not many. Perhaps the lack of data resources, non-disclosure of sources by research groups, and decentralized research communities are the main reasons for Persian's current state. Of course, it should be noted that some of the research groups concern about this matter and share their results, thoughts, and resources with others, but still, we need more. NLI (known as recognizing textual entailment) resources for Persian are vital for every semantic, extraction, and inference system. I was so excited when FarsTail, as the first NLI dataset for Persian, was released. I used this dataset to train a Sentence-Transformer model (using ParsBERT) as a basis for other applications like Semantic Search, Clustering, Information Extraction, Summarization, Topic Modeling, and some others. However, the model could be achieved remarkable results on recognizing entailment (81.71%) in contrast to what they mentioned in their paper (78.13%), still not adequate for NLI applications. I dug in the official paper of Sentence-Transformer Reimers and Gurevych, 2019. I found that it used the Wikipedia-Triplet-Sections, introduced by Dor et al., 2018, to train the SBERT for recognizing entailment tasks. Dor et al., 2018, presume that sentences in the same section are thematically closer than sentences in different sections. They presented the anchor () and the positive () example from the same section, while the negative example () comes from a separate section of the same article. They designated the following steps to generate this sentences-triplet dataset (In each rule, I would specify whether to use the principal or not): Exploit Wikipedia partitioning into sections and paragraphs, using OpenNLP for sentence extraction. Apply the following rules and filters to reduce noise and to create a high-quality dataset, ‘triplets sent: The maximal distance between the intra-section sentences is limited to three paragraphs. (Change this rule into two terms, inner and outer parts. The sentences from the outer part () must be with a distance of at least two sections. The sentences from the inner part () must be a distance of at most two paragraphs.) Sentences with less than 5 or more than 50 tokens are filtered out. (Change this relation into 10 &lt; length of word tokens &lt; 130.) The first and the ”Background” sections are removed due to their general nature. (Do the same.) The following sections are removed: "External links", "Further reading", "References", "See also", "Notes","Citations" and "Authored books". These sections usually list a set of items rather than discuss a specific subtopic of the article’s title. (Add a few more filters: محتویات-پانویس-منابع-منابع و پانویس-جستارهای وابسته-پیوند به بیرون-یادداشت‌ها-یادداشت‌ ها-جوایز-نگارخانه-روابط‌خارجی-روابط خارجی-کتاب‌شناسی-کتاب‌ شناسی-فیلم‌شناسی-فیلم ‌شناسی-دست‌اندرکاران-دست‌اندر کاران-دست‌ اندر کاران-فروش‌های برگزیدهٔ آلبوم-فروش‌ های برگزیدهٔ آلبوم-فروش‌ های برگزیده آلبوم-نمودارهای فروش-نمودار های فروش-فهرست آهنگ‌ها-فهرست آهنگ‌ ها-اعضا-ترانه‌شناسی-ترانه‌ شناسی-نگارخانه-بازیگران-پروژه‌های مشابه-پروژه‌ های مشابه) Only articles with at least five remaining sections are considered to focus on articles with rich enough content. (Skip this rule.) Reimers and Gurevych, 2019 use the dataset with a Triplet Objective to train the SBERT. \(\begin{equation} \max \left(\left\|s_{a}-s_{p}\right\|-\left\|s_{a}-s_{n}\right\| \epsilon, 0\right) \end{equation}\) <br> <em>Eq 1: Triplet Objective Function, try to minimize the above loss function.</em> Tips: SBERT adds a pooling operation to the output of BERT / RoBERTa to derive a fixed-sized sentence embedding. They experimented with three pooling strategies: Using the production of the CLS token. Computing the mean of all output vectors (Mean-Strategy). Computing a max-over-time of the output vectors (Max-Strategy). In this case, I use <em>Mean-Strategy</em>. <strong>Dataset Information</strong> <strong>Version 1.0.0</strong> Wikipedia Section Sentences Wiki Triplet Wiki D/Similar
description: {"references": ["Learning Thematic Similarity Metric from Article Sections Using Triplet Networks, 10.18653/v1/P18-2009", "ParsBERT: Transformer-based Model for Persian Language Understanding, arXiv:2005.12515", "FarsTail: A Persian Natural Language Inference Dataset, arXiv:2009.08820", "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, arXiv:1908.10084"]}