Repository: blevesearch/bleve
Pull request #2103 (2024-11-21 13:28:16)

Move PartialMatch field to Explanation struct and update related logic
- Previously, the `PartialMatch` field was returned for every hit, but this caused confusion in complex queries involving disjunctions and match queries. As a result, we moved `PartialMatch` to the score explanation, where each subquery's explanation will include its own `PartialMatch`. This field is set only if the query uses the DisjunctionSearcher or scorer.

abhinavdangeti: So just so we're clear here - partial_match will be set if an explicit disjunction were used and all its components did not match yes?
And also separately, if a match query resolves to a disjunction - and if some of the components did not match, we'll now be setting partial_match - correct?
CascadingRadium: - A match query will be a disjunction query over all the terms, here if the document has all 
  the terms then partial_match will be set to true else it will be set to false.
- For all other queries partial_match will never show up in the search result as its `omitempty` and will be simply excluded from the search result.
abhinavdangeti: Ok, then I disagree with your proposal here - we should be returning a partial_match when a disjunction's component doesn't match as well.
CascadingRadium: like in a disjunction query ?
abhinavdangeti: > like when the top-most query is a disjunction query ?

Ya.
CascadingRadium: Actually, this approach has the limitation with disjunction + match queries
like so 

```
{
  "disjuncts":[
    {
      "match":"termA termB",
    },
    {
      "match":"SONY PS5",
      "prefix_length":0,
      "fuzziness":1
    },
  ]
}
```

where a single PartialMatch in the searchResult would lead to confusion regarding which component was partially matched (matchQuery1 / matchQuery2 / outerDisjuncts)

i will propose a new solution where the partialMatch is reflected in the score explanation, where each subQuery's expl will have the partialMatch attribute

Patch:
diff --git a/docs/scoring.md b/docs/scoring.md
index ddcfcb2c..2b4b747e 100644
--- a/docs/scoring.md
+++ b/docs/scoring.md
@@ -1,5 +1,6 @@
 # Scoring models for document hits
 
+* Search is performed on a collection fields using compound queries such as conjunction/disjunction/boolean etc. However, the scoring itself is done independently for each field and then aggregated to get the final score for a document hit.
 * Default scoring scheme for document hits involving text hits: `tf-idf`.
 * Nearest-neighbor/vector hits scoring depends on chosen `knn distance` metric, highlighted [here](https://github.com/blevesearch/bleve/blob/master/docs/vectors.md#supported).
 * Hybrid search scoring will combine `tf-idf` scores with `knn distance` numbers.
@@ -7,4 +8,81 @@
 
 ## BM25
 
-`<wip>`
+When it comes to scoring a document hit for a specific field, BM25 scoring mechanism requires the following stats:
+* fieldLen - The number of analyzed terms in the current document's field.
+* avgFieldLen - The average number of analyzed terms in the field across all the documents.
+* docTotal - The total number of documents in the index.
+* docTerm - The total number of documents containing the query term within the index.
+
+
+The scoring formula followed in BM25 is
+
+```math
+\sum_{i}^n IDF(q_i) {{f(q_i,D) * (k1 + 1)}\over{f(q_i,D) + k1 * (1-b+b*{{fieldLen}\over{avgFieldLen}})}}
+```
+
+$IDF(q_i)$ here refers to Inverse Document Frequency talks about how rare (and hence rich in information) is a particular query term $`q_i`$ across all the documents in the index, which is calculated as
+```math
+    \ln(1 + {{docTotal - docTerm + 0.5}\over{docTerm + 0.5}})
+```
+
+Coming back to the BM25 scoring, $f(q_i,D)$ refers to the frequency of the query term in document $D$. The entire equation has certain multipliers 
+* $k1$ - helps in controlling the saturation of the score with respect to query term in a document. Basicaly if the query term's frequency is too high, the score value gets saturated and doesn't increase beyond a certain point.
+* $b$ - controls the extent to which the $fieldLen$ normalizes the term's frequency. 
+
+### How to enable and use BM25
+
+Bleve v2.5.0 updated the `indexMapping` construct with the concept of `scoringModel`. This is a global (meaning applicable to all the fields) setting which drives which scoring algorithm to apply while scoring the document hits. Supported scoring models are defined [here](https://github.com/blevesearch/bleve_index_api/blob/f54d76f0a71a838837159aa44ced0404bb6ec25f/indexing_options.go#L27)
+
+For instance, while defining the index mapping for the data model that's been decided by the user, following snippet can be referred to enable BM25
+
+```go
+indexMapping := bleve.NewIndexMapping()
+indexMapping.TypeField = "type"
+indexMapping.DefaultAnalyzer = "en"
+indexMapping.ScoringModel = index.BM25Scoring
+```
+
+During search time there's explicit change involved, unless the user wants to perform a global scoring.
+
+### Global Scoring
+
+Let's say that the user has a dataset which is quite large (let's say 3 million) and to have good throughput, they create 3 shards (with the same index mapping) for the "index". Each of these shards can be `bleve.Index` type and while performing a search over the entire dataset, a `bleve.IndexAlias` can be created over which a search can be performed. This parallelizes things pretty good, both on the indexing path and the search path.
+
+The concept of global scoring is applicable when the index is "sharded" (similar to above situation). This is because each index has data which is disjoint, and thereby while performing the scoring on document hits on each of them, the value of stats is not complete at a global level, since we're doing a search over the entire dataset using the `bleve.IndexAlias`. For eg: `docTotal` value while scoring the document hits would be 1 million which is incorrect at a global level. 
+
+So in order to keep the scoring roughly same across varying count of the number of shards involved, we provide a mechanism to enable "global scoring". In this type of search, an initial roundtrip is performed to gather and aggregate the stats necessary for the scoring mechanism and in the second phase, the actual search is performed. So naturally this comes at a cost of latency. As a reference here's how the user can go about with it
+
+```go
+multiPartIndex := bleve.NewIndexAlias(shard1, shard2)
+// set the alias with the same index mapping which both the shards use.
+err = multiPartIndex.SetIndexMapping(indexMapping)
+if err != nil {
+    return err
+}
+
+ctx := context.Background()
+ctx = context.WithValue(ctx, search.SearchTypeKey, search.GlobalScoring)
+
+res, err := multiPartIndex.SearchInContext(ctx, searchRequest)
+```
+
+A note here is that, this would only matter if the relative order of the document hits vary quite a bit (vs single shard case). This would be possible when the shard count increases quite a bit, in low doc count situations or if there is a heavy skew in the data distribution amongst the shards for some reason. Ideally the shards are created when the data is quite large and each of them index same amount of data - in which case the scores won't fluctuate much to affect the relative hit order and the user can choose to avoid the global scoring mechanism altogether. 
+
+## TF-IDF
+
+TF-IDF is the default scoring mechanism involved (for backward compatibility reasons) and requires no change from the user at index or search time to avail it. 
+
+The scoring formula involved is
+
+```math
+    \sum_{i}^n f(q_i, D) * {{1}\over{\sqrt{fieldLen}}} * IDF(q_i)
+```
+
+where $IDF(q_i)$ is
+
+```math
+    1 + {{docTotal}\over{1 + docTerm}}
+```
+
+Note: TF-IDF formula doesn't accomodate logic for score saturation due to term frequency or fieldLen. So, it's recommended to use BM25 scoring by explicity setting it in the index mapping.
\ No newline at end of file
diff --git a/docs/synonyms.md b/docs/synonyms.md
index f27e3d63..0777debd 100644
--- a/docs/synonyms.md
+++ b/docs/synonyms.md
@@ -1,5 +1,180 @@
-# Synonyms search
+# Synonym search
 
-* *v2.5.0* (and after) will come with support for **synonym definitions indexing and search**.
+* *v2.5.0* (and after) will come with support for **synonym definition indexing and search**.
+* We've achieved this by embedding synonym indexes within our bleve (scorch) indexes.
+* Usage of zap file format: [v16](https://github.com/blevesearch/zapx/blob/master/zap.md). Here we co-locate text, vector and synonym indexes as neighbors within segments, continuing to conform to the segmented architecture of *scorch*.
 
-`<wip>`
+## Supported
+
+* Indexing `Synonym Definitions` allows specifying equivalent terms that will be used to construct the synonym index. There are currently two types of `Synonym Definitions` supported:
+
+    1. Equivalent Mapping:
+
+        In this type, all terms in the *synonyms* list are considered equal and can replace one another. Any of these terms can match a query or document containing any other term in the group, ensuring full synonym coverage.
+
+        ```json
+        {
+            "synonyms": [
+                "tranquil",
+                "peaceful",
+                "calm",
+                "relaxed",
+                "unruffled"
+            ]
+        }
+        ```
+
+    2. Explicit Mapping:
+
+        In this mapping, only the terms in the *input* list ("blazing") will have the terms in *synonyms* as their synonyms. The input terms are not equivalent to each other, and the synonym relationship is explicitly directional, applying only from the *input* to the *synonyms*.
+
+        ```json
+        {
+            "input": [
+                "blazing"
+            ],
+            "synonyms": [
+                "intense",
+                "radiant",
+                "burning",
+                "fiery",
+                "glowing"
+            ]
+        }
+        ```
+
+* The addition of `Synonym Sources` in the index mapping enables associating a set of `synonym definitions` (called a `synonym collection`) with a specific analyzer. This allows for preprocessing of terms in both the *input* and *synonyms* lists before the synonym index is created. By using an analyzer, you can normalize or transform terms (e.g., case folding, stemming) to improve synonym matching.
+
+    ```json
+    {
+        "analysis": {
+            "synonym_sources": {
+                "english": {
+                    "collection": "en_thesaurus",
+                    "analyzer": "en"
+                },
+                "german": {
+                    "collection": "de_thesaurus",
+                    "analyzer": "de"
+                }
+            }
+        }
+   }
+   ```
+
+    There are two `synonym sources` named "english" and "german," each associated with its respective `synonym collection` and analyzer. In any text field mapping, a `synonym source` can be specified to enable synonym expansion when the field is queried. The analyzer of the synonym source must match the analyzer of the field mapping to which it is applied.
+
+* Any text-based Bleve query (e.g., match, phrase, term, fuzzy, etc.) will use the `synonym source` (if available) for the queried field to expand the search terms using the thesaurus created from user-defined synonym definitions. The behavior for specific query types is as follows:
+
+    1. Queries with `fuzziness` parameter: For queries like match, phrase, and match-phrase that support the `fuzziness` parameter, the queried terms are fuzzily matched with the thesaurus's LHS terms to generate candidate terms. These terms are then combined with the results of fuzzy matching against the field dictionary, which contains the terms present in the queried field.
+
+    2. Wildcard, Regexp, and Prefix queries: These queries follow a similar approach. First, the thesaurus is used to expand terms (e.g., LHS terms that match the prefix or regex). The resulting terms are then combined with candidate terms from dictionary expansion.
+
+## Indexing
+
+Below is an example of using the Bleve API to define synonym sources, index synonym definitions, and associate them with a text field mapping:
+
+```go
+// Define a document to be indexed.
+doc := struct {
+    Text string `json:"text"`
+}{
+    Text: "hardworking employee",
+}
+
+// Define a synonym definition where "hardworking" has equivalent terms.
+synDef := &bleve.SynonymDefinition{
+    Synonyms: []string{
+        "hardworking",
+        "industrious",
+        "conscientious",
+        "persistent",
+    },
+}
+
+// Define the name of the `synonym collection`.
+// This collection groups multiple synonym definitions.
+synonymCollection := "collection1"
+
+// Define the name of the `synonym source`.
+// This source will be associated with specific field mappings.
+synonymSourceName := "english"
+
+// Define the analyzer to process terms in the synonym definitions.
+// This analyzer must match the one applied to the field using the synonym source.
+analyzer := "en"
+
+// Configure the synonym source by associating it with the synonym collection and analyzer.
+synonymSourceConfig := map[string]interface{}{
+    "collection": synonymCollection,
+    "analyzer":   analyzer,
+}
+
+// Create a new index mapping.
+bleveMapping := bleve.NewIndexMapping()
+
+// Add the synonym source configuration to the index mapping.
+err := bleveMapping.AddSynonymSource(synonymSourceName, synonymSourceConfig)
+if err != nil {
+    panic(err)
+}
+
+// Create a text field mapping with the specified analyzer and synonym source.
+textFieldMapping := bleve.NewTextFieldMapping()
+textFieldMapping.Analyzer = analyzer
+textFieldMapping.SynonymSource = synonymSourceName
+
+// Associate the text field mapping with the "text" field in the default document mapping.
+bleveMapping.DefaultMapping.AddFieldMappingsAt("text", textFieldMapping)
+
+// Create a new index with the specified mapping.
+index, err := bleve.New("example.bleve", bleveMapping)
+if err != nil {
+    panic(err)
+}
+
+// Index the document into the created index.
+err = index.Index("doc1", doc)
+if err != nil {
+    panic(err)
+}
+
+// Check if the index supports synonym indexing and add the synonym definition.
+if synIndex, ok := index.(bleve.SynonymIndex); ok {
+    err = synIndex.IndexSynonym("synDoc1", synonymCollection, synDef)
+    if err != nil {
+        panic(err)
+    }
+} else {
+    // If the index does not support synonym indexing, raise an error.
+    panic("expected synonym index")
+}
+```
+
+## Querying
+
+```go
+// Query the index created above.
+// Create a match query for the term "persistent".
+query := bleve.NewMatchQuery("persistent")
+
+// Specify the field to search within, in this case, the "text" field.
+query.SetField("text")
+
+// Create a search request with the query and enable explanation to understand how results are scored.
+searchRequest := bleve.NewSearchRequest(query)
+searchRequest.Explain = true
+
+// Execute the search on the index.
+searchResult, err := index.Search(searchRequest)
+if err != nil {
+    // Handle any errors that occur during the search.
+    panic(err)
+}
+
+// The search result will contain one match: "doc1". This document includes the term "hardworking", 
+// which is a synonym for the queried term "persistent". The synonym relationship is based on 
+// the user-defined thesaurus associated with the index.
+// Print the search results, which will include the explanation for the match.
+fmt.Println(searchResult)
+```
diff --git a/search/explanation.go b/search/explanation.go
index b1ac29aa..92405001 100644
--- a/search/explanation.go
+++ b/search/explanation.go
@@ -30,9 +30,10 @@ func init() {
 }
 
 type Explanation struct {
-	Value    float64        `json:"value"`
-	Message  string         `json:"message"`
-	Children []*Explanation `json:"children,omitempty"`
+	Value        float64        `json:"value"`
+	Message      string         `json:"message"`
+	PartialMatch bool           `json:"partial_match,omitempty"`
+	Children     []*Explanation `json:"children,omitempty"`
 }
 
 func (expl *Explanation) String() string {
diff --git a/search/scorer/scorer_disjunction.go b/search/scorer/scorer_disjunction.go
index fe319bbe..b3e96ddc 100644
--- a/search/scorer/scorer_disjunction.go
+++ b/search/scorer/scorer_disjunction.go
@@ -69,7 +69,7 @@ func (s *DisjunctionQueryScorer) Score(ctx *search.SearchContext, constituents [
 		ce := make([]*search.Explanation, 2)
 		ce[0] = rawExpl
 		ce[1] = &search.Explanation{Value: coord, Message: fmt.Sprintf("coord(%d/%d)", countMatch, countTotal)}
-		newExpl = &search.Explanation{Value: newScore, Message: "product of:", Children: ce}
+		newExpl = &search.Explanation{Value: newScore, Message: "product of:", Children: ce, PartialMatch: countMatch != countTotal}
 	}
 
 	// reuse constituents[0] as the return value
diff --git a/search/search.go b/search/search.go
index 8cc5115d..066f8858 100644
--- a/search/search.go
+++ b/search/search.go
@@ -167,13 +167,6 @@ type DocumentMatch struct {
 	// results are completed
 	FieldTermLocations []FieldTermLocation `json:"-"`
 
-	// used to indicate if this match is a partial match
-	// in the case of a disjunction search
-	// this means that the match is partial because
-	// not all sub-queries matched
-	// if false, all the sub-queries matched
-	PartialMatch bool `json:"partial_match,omitempty"`
-
 	// used to indicate the sub-scores that combined to form the
 	// final score for this document match.  This is only populated
 	// when the search request's query is a DisjunctionQuery
diff --git a/search/searcher/search_disjunction_heap.go b/search/searcher/search_disjunction_heap.go
index 89bcd498..3da876bd 100644
--- a/search/searcher/search_disjunction_heap.go
+++ b/search/searcher/search_disjunction_heap.go
@@ -218,9 +218,7 @@ func (s *DisjunctionHeapSearcher) Next(ctx *search.SearchContext) (
 				rv = s.scorer.ScoreAndExplBreakdown(ctx, s.matching, s.matchingIdxs, nil, s.numSearchers)
 			} else {
 				// score this match
-				partialMatch := len(s.matching) != len(s.searchers)
 				rv = s.scorer.Score(ctx, s.matching, len(s.matching), s.numSearchers)
-				rv.PartialMatch = partialMatch
 			}
 		}
 
diff --git a/search/searcher/search_disjunction_slice.go b/search/searcher/search_disjunction_slice.go
index 81b00cc2..fadfa59c 100644
--- a/search/searcher/search_disjunction_slice.go
+++ b/search/searcher/search_disjunction_slice.go
@@ -230,9 +230,7 @@ func (s *DisjunctionSliceSearcher) Next(ctx *search.SearchContext) (
 				rv = s.scorer.ScoreAndExplBreakdown(ctx, s.matching, s.matchingIdxs, s.originalPos, s.numSearchers)
 			} else {
 				// score this match
-				partialMatch := len(s.matching) != len(s.searchers)
 				rv = s.scorer.Score(ctx, s.matching, len(s.matching), s.numSearchers)
-				rv.PartialMatch = partialMatch
 			}
 		}
 


Test patch:
diff --git a/search_test.go b/search_test.go
index fdfaa3ef..2271018a 100644
--- a/search_test.go
+++ b/search_test.go
@@ -1266,6 +1266,7 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 	mq1.SetField("description")
 
 	sr := NewSearchRequest(mq1)
+	sr.Explain = true
 	res, err := idx.Search(sr)
 	if err != nil {
 		t.Fatal(err)
@@ -1274,11 +1275,17 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 		t.Errorf("Expected 2 results, but got: %v", res.Total)
 	}
 	for _, hit := range res.Hits {
-		if hit.ID == "doc1" && hit.PartialMatch {
-			t.Errorf("Expected doc1 to be a full match")
-		}
-		if hit.ID == "doc2" && !hit.PartialMatch {
-			t.Errorf("Expected doc2 to be a partial match")
+		switch hit.ID {
+		case "doc1":
+			if hit.Expl.PartialMatch {
+				t.Errorf("Expected doc1 to be a full match")
+			}
+		case "doc2":
+			if !hit.Expl.PartialMatch {
+				t.Errorf("Expected doc2 to be a partial match")
+			}
+		default:
+			t.Errorf("Unexpected document ID: %s", hit.ID)
 		}
 	}
 
@@ -1288,6 +1295,7 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 	mq2.SetFuzziness(2)
 
 	sr = NewSearchRequest(mq2)
+	sr.Explain = true
 	res, err = idx.Search(sr)
 	if err != nil {
 		t.Fatal(err)
@@ -1296,11 +1304,17 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 		t.Errorf("Expected 2 results, but got: %v", res.Total)
 	}
 	for _, hit := range res.Hits {
-		if hit.ID == "doc1" && !hit.PartialMatch {
-			t.Errorf("Expected doc1 to be a partial match")
-		}
-		if hit.ID == "doc2" && hit.PartialMatch {
-			t.Errorf("Expected doc2 to be a full match")
+		switch hit.ID {
+		case "doc1":
+			if !hit.Expl.PartialMatch {
+				t.Errorf("Expected doc1 to be a partial match")
+			}
+		case "doc2":
+			if hit.Expl.PartialMatch {
+				t.Errorf("Expected doc2 to be a full match")
+			}
+		default:
+			t.Errorf("Unexpected document ID: %s", hit.ID)
 		}
 	}
 	// Test 3 - Two Docs hits, both full match
@@ -1308,6 +1322,7 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 	mq3.SetField("description")
 
 	sr = NewSearchRequest(mq3)
+	sr.Explain = true
 	res, err = idx.Search(sr)
 	if err != nil {
 		t.Fatal(err)
@@ -1316,11 +1331,17 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 		t.Errorf("Expected 2 results, but got: %v", res.Total)
 	}
 	for _, hit := range res.Hits {
-		if hit.ID == "doc1" && hit.PartialMatch {
-			t.Errorf("Expected doc1 to be a full match")
-		}
-		if hit.ID == "doc2" && hit.PartialMatch {
-			t.Errorf("Expected doc2 to be a full match")
+		switch hit.ID {
+		case "doc1":
+			if hit.Expl.PartialMatch {
+				t.Errorf("Expected doc1 to be a full match")
+			}
+		case "doc2":
+			if hit.Expl.PartialMatch {
+				t.Errorf("Expected doc2 to be a full match")
+			}
+		default:
+			t.Errorf("Unexpected document ID: %s", hit.ID)
 		}
 	}
 	// Test 4 - Two Docs hits, both partial match
@@ -1328,6 +1349,7 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 	mq4.SetField("description")
 
 	sr = NewSearchRequest(mq4)
+	sr.Explain = true
 	res, err = idx.Search(sr)
 	if err != nil {
 		t.Fatal(err)
@@ -1336,11 +1358,17 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 		t.Errorf("Expected 2 results, but got: %v", res.Total)
 	}
 	for _, hit := range res.Hits {
-		if hit.ID == "doc1" && !hit.PartialMatch {
-			t.Errorf("Expected doc1 to be a partial match")
-		}
-		if hit.ID == "doc2" && !hit.PartialMatch {
-			t.Errorf("Expected doc2 to be a partial match")
+		switch hit.ID {
+		case "doc1":
+			if !hit.Expl.PartialMatch {
+				t.Errorf("Expected doc1 to be a full match")
+			}
+		case "doc2":
+			if !hit.Expl.PartialMatch {
+				t.Errorf("Expected doc2 to be a full match")
+			}
+		default:
+			t.Errorf("Unexpected document ID: %s", hit.ID)
 		}
 	}
 
@@ -1350,6 +1378,7 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 	mq5.SetOperator(1)
 
 	sr = NewSearchRequest(mq5)
+	sr.Explain = true
 	res, err = idx.Search(sr)
 	if err != nil {
 		t.Fatal(err)
@@ -1357,7 +1386,9 @@ func TestMatchQueryPartialMatch(t *testing.T) {
 	if res.Total != 1 {
 		t.Errorf("Expected 1 result, but got: %v", res.Total)
 	}
-	if res.Hits[0].ID == "doc2" || res.Hits[0].PartialMatch {
+	hit := res.Hits[0]
+	fmt.Println(hit.Expl, hit.ID)
+	if hit.ID != "doc1" || hit.Expl.PartialMatch {
 		t.Errorf("Expected doc1 to be a full match")
 	}
 }