Fine-tuning a
multi-language model for
information retrieval using
the Solr Manual
By Elpidio Gonzalez

SummaryWe
ﬁne-tunedapre-trained
multilanguage SBERT model on theSolr Manual to test the feasibility ofhaving a semantic search engine
capable of retrieving documents
written in a particular language, inthis case, English, using non-english
queries.

Agenda

●
●
●
●
●
●
●
●

Semantic Search Overview
Pretrained Models
Solr Manual as a Dataset
Fine-Tuning a Model
Model Evaluation
Solr Schema and Indexing
Improvements
DemoSemantic Search

Semantic search is a data searching
technique that aims to understandthe meaning and context of words
and phrases in a search query, as
opposed to just matching keywords.
This means that it can provide more
relevant and accurate results to auser's query, even if the search terms
used aren't an exact match to thecontent being searched.

Document Representation

Semantic Search in SolrIn the context of Solr, Semantic Search if referred to as Neural Search.
The ﬁrst implementation was contributed to the Solr open source community
by Sease in January 2022 thanks to the work of Alessandro Benedetti and Elia
Porciani.
This implementation relies on the Apache Lucene implementation for
K-nearest neighbor search to provide:●A Dense Vector Field type : To store text embeddings●A KNN Query Parser to run knn search on a target vector ﬁeld : Tocompare query - document similarity.

Pretrained Models

Pretrained text models are deep learningmodels that have been pre-trained on large
amounts of text data to learn the underlying
patterns and relationships between words
and phrases. These models are trained using
techniques such as neural networks, and aredesigned to capture the meaning and context
of text data. As such, they are the perfect tool
to calculate a document's text embeddings.
An example of such models is SBERT.

Pre-trained Model Advantages●●●●

Time and Resource Savings
High Performance
Transfer Learning
Accessible and Open SourcePre-trained multi-lingual model

For our particular task we used thesentence-transformers/paraphrase-multilingual-mpnet-base-v2
model available on Hugging Face.
This particular model was trained using over a dozen datasets of parallel data
(multiple languages), notably WikiMatrix, a dataset of 135M Parallel Sentences in
1,620 Language Pairs from Wikipedia.
This model maps sentences & paragraphs to a 768 dimensional dense vector spaceand can be used for tasks like clustering or semantic search.

Solr Manual

The Solr manual is a comprehensive
guide that provides detailed
information on how to install,
conﬁgure, use, and maintain theSolr search platform.

It is the ofﬁcial Solr documentation,
written and published by Solr
committers, presented in adocformat.

Adoc Format

The AsciiDoc (adoc) format is a lightweight markup language that is used towrite technical documentation.
Some of the features of the AsciiDoc format include:●●●●●Headings and subheadings
Lists and tables
Links and cross-references
Images and diagrams
Source code blocks with syntax highlighting

Cleaning the Documents● Remove Code Snippets, Hyperlinks and Images○ Code snippets ([source, ...])○ Links (xref)○ Images (image:: )● Remove License Notice● Leave Tables● Use heading and subheading syntax to split documents.
○ Heading (=Heading)○ Sub-heading 1 (==Heading)Splitting the ManualThe root document 'Field Type Deﬁnitions and Properties would be split into:●●Field Type Deﬁnitions and Properties / Field Type Deﬁnitions in theSchema
Field Type Deﬁnitions and Properties / Field Type Properties / GeneralProperties...
●

Field Type Deﬁnitions and Properties / Choosing Appropriate Numeric
Types

…

This data processing resulted in a total of 2,972 documents.

FacetingFine-TuningQuery Generation
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information
Retrieval Models proposed using synthetic queries for unsupervised
domain-adaptation approach for dense retrieval models.
We generated 4 extra queries per document and used the original title as a ﬁfthone.

Translation

こんにちは●Target Languages:
English and Japanese●Total Training Samples:
29,720Fine-tuning a Bi-encoder model
Loss Function

CosineSimilarity?
MultipleNegativesRanking?
Triplet?

Multiple Negatives Ranking Loss

Multiple Negatives Ranking Loss

For each query (ai) the MultipleNegativesRankingLoss
class will create negative samples by randomly choosing
non-positive documents associated with the otherqueries bj (j != i). So , a batch of samples in a training stepwould look like:(a1 - b1 )(a2 - b2 )(a3 - b3 )…
(an- bn )(a1 - b1 )(a1 - b2 )(a1 - b3 )…
(a1- bn )(a2 - b1 )(a2 - b2 )(a2 - b3 )…
(a2- bn )…

(an - b1 )(an - b2 )(an - b3 )…
(an- bn )Hard negatives can be introduced by structuring thesamples like this: (a1 - b1 - n1 )

Evaluation

To evaluate our model we use the InformationRetrievalEvaluator in the SentenceTransformers library. Itprovides a way to evaluate the performance of a model on an information retrieval task. It computes
standard IR metrics such as recall, precision, and mean average precision (MAP).
●
●
●
●
●

Input Data : A list of queries and a corresponding list of documents for each query.
Model Inference: The model is used to encode the queries and documents into ﬁxed-length
vector representations.
Similarity Scoring: A similarity function such as cosine similarity or dot product. The similarity
scores are used to rank the documents.
Metrics Calculation: The ranked list of documents for each query is compared to the groundtruth relevance labels to calculate metrics such as recall, precision, and MAP.
Output: A dictionary containing the computed metrics, including recall, precision, and MAP.

Results
MetricValue

MetricValue

Accuracy @ 10.890

Recall @ 10.890

Accuracy @ 30.993

Recall @ 30.993

Accuracy @ 50.999

Recall @ 50.999

Precision @ 10.890

NDCG @ 10

0.956

Precision @ 30.331

MRR @ 100.941

Precision @ 5

0.199

mAP@ 1000.941

The small print about the results…

The results above look very good because the evaluation is run on the entire corpus. We didn't
work with a train/test split because …
1.

The manual will not change drastically over time. The existing documents might change a bit,but we can consider them pretty much static.
New documents will deﬁnitely be added over time, but...

2.

Re-training the model using new data is not a computationally expensive operation. The corpus
is very small. Also, we can re-use the generated queries and generate new ones only for
any new additions to the corpus. Given that the training can be ﬁnished in under 5 hours(20 epochs @ Google Colab), we can update the model every time a new Solr version
comes out.

Basically, we don't care about overﬁtting the model (high variance, low bias), because the model
won't work with unseen data at all.

Improvements

1. Symmetric Vs. Asymmetric Search Models
2. Better Synthetic Queries
3. Real User QueriesApache Solr Dense Vectors

The dense vector ﬁeld gives the possibility of indexing and searching dense vectors of ﬂoat
elements. e.g. [1.0, 2.5, 3.7, 4.1]
Here’s how DenseVectorFieldshould be conﬁgured in the schema:<fieldType name="knn_vector"
class="solr.DenseVectorField"
vectorDimension="4"
similarityFunction="cosine"/><field name="myVector" type="knn_vector" indexed="true"stored="true"/>Apache Solr Dense Vectors

Parameter Name

Required

vectorDimension

TruesimilarityFunction False

Default

Description

Accepted values

The dimension of the dense
vector to pass in.

Integer < = 1024

euclidean Vector similarity function; used insearch to return top K mostsimilar vectors to a target vector.

euclidean, dot_product orcosine.

knn Query Parser

Parameter Name

Requiredf

True

topK

False

Default

Description

The DenseVectorField to search in.

10

How many k-nearest results to return.

&q={!knn f=myVector topK=10}[1.0, 2.0, 3.0, 4.0]

DemoThank you!