Repository: yeraydiazdiaz/lunr.py

Issue:
How to disable stop_word_filter trimmer for specific fields
Hello, 

I'm building [search system](https://pydoctor--500.org.readthedocs.build/en/500/api/index.html) and generate a index with `lunr.py` (actually searching with `lunr.js`) and I would like to disable the `stop_word_filter` for fields that are not actually text, like object names. Because objects can be named `If` or `For` (in the AST library for instance). 

Do you know if I can reach such behaviour ?

Thanks

Discussion:
tristanlatr: Static checks are fixed now! 
codecov-commenter: # [Codecov](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz) Report
> Merging [#109](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz) (5764eb8) into [master](https://codecov.io/gh/yeraydiazdiaz/lunr.py/commit/b9533caab2fd68513098cd7eaa9e011d7b15fedc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz) (b9533ca) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109/graphs/tree.svg?width=650&height=150&src=pr&token=9bkIWilQ5B&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz)](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz)

```diff
@@           Coverage Diff           @@
##           master     #109   +/-   ##
=======================================
  Coverage   98.34%   98.35%           
=======================================
  Files          23       23           
  Lines        1089     1095    +6     
  Branches      199      218   +19     
=======================================
+ Hits         1071     1077    +6     
  Misses         10       10           
  Partials        8        8           
```


| [Impacted Files](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz) | Coverage Δ | |
|---|---|---|
| [lunr/builder.py](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz#diff-bHVuci9idWlsZGVyLnB5) | `100.00% <100.00%> (ø)` | |
| [lunr/pipeline.py](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz#diff-bHVuci9waXBlbGluZS5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz). Last update [b9533ca...5764eb8](https://codecov.io/gh/yeraydiazdiaz/lunr.py/pull/109?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yeray+Diaz+Diaz).


Patch:
diff --git a/docs/customisation.md b/docs/customisation.md
index 6106f1d..ed187a0 100644
--- a/docs/customisation.md
+++ b/docs/customisation.md
@@ -41,6 +41,23 @@ idx = lunr(ref="id", fields=("title", "body"), documents=documents, builder=buil
 Note pipeline functions take the token being processed, its position in the
 token list, and the token list itself.
 
+## Skip a pipeline function for specific field names
+
+The `Pipeline.skip()` method allows you to skip a pipeline function for specific field names.
+This example skips the `stop_word_filter` pipeline function for the field `fullName`.
+
+```python
+from lunr import lunr, get_default_builder, stop_word_filter
+
+documents = [...]
+
+builder = get_default_builder()
+
+builder.pipeline.skip(stop_word_filter.stop_word_filter, ["fullName"])
+
+idx = lunr(ref="id", fields=("fullName", "body"), documents=documents, builder=builder)
+```
+
 ## Token meta-data
 
 Lunr.py `Token` instances include meta-data information which can be used in
diff --git a/lunr/builder.py b/lunr/builder.py
index 7a4cf39..6f319b5 100644
--- a/lunr/builder.py
+++ b/lunr/builder.py
@@ -144,7 +144,7 @@ class Builder:
             extractor = field.extractor
             field_value = doc[field_name] if extractor is None else extractor(doc)
             tokens = Tokenizer(field_value)
-            terms = self.pipeline.run(tokens)
+            terms = self.pipeline.run(tokens, field_name)
             field_ref = FieldRef(doc_ref, field_name)
             field_terms = defaultdict(int)
 
diff --git a/lunr/pipeline.py b/lunr/pipeline.py
index fc51380..0e0ab4e 100644
--- a/lunr/pipeline.py
+++ b/lunr/pipeline.py
@@ -1,5 +1,6 @@
+from collections import defaultdict
 import logging
-from typing import Callable, Dict
+from typing import Callable, Dict, List, Set
 
 from lunr.exceptions import BaseLunrException
 from lunr.token import Token
@@ -16,7 +17,8 @@ class Pipeline:
     registered_functions: Dict[str, Callable] = {}
 
     def __init__(self):
-        self._stack = []
+        self._stack: List[Callable] = []
+        self._skip: Dict[Callable, Set[str]] = defaultdict(set)
 
     def __len__(self):
         return len(self._stack)
@@ -104,10 +106,27 @@ class Pipeline:
         except ValueError:
             pass
 
-    def run(self, tokens):
-        """Runs the current list of functions that make up the pipeline against
-        the passed tokens."""
+    def skip(self, fn: Callable, field_names: List[str]):
+        """
+        Make the pipeline skip the function based on field name we're processing.
+
+        This relies on passing the field name to Pipeline.run().
+        """
+        self._skip[fn].update(field_names)
+
+    def run(self, tokens, field_name=None):
+        """
+        Runs the current list of functions that make up the pipeline against
+        the passed tokens.
+
+        :param tokens: The tokens to process.
+        :param field_name: The name of the field these tokens belongs to, can be ommited.
+            Used to skip some functions based on field names.
+        """
         for fn in self._stack:
+            # Skip the function based on field name.
+            if field_name and field_name in self._skip[fn]:
+                continue
             results = []
             for i, token in enumerate(tokens):
                 # JS ignores additional arguments to the functions but we
@@ -127,7 +146,11 @@ class Pipeline:
     def run_string(self, string, metadata=None):
         """Convenience method for passing a string through a pipeline and
         getting strings out. This method takes care of wrapping the passed
-        string in a token and mapping the resulting tokens back to strings."""
+        string in a token and mapping the resulting tokens back to strings.
+
+        .. note:: This ignores the skipped functions since we can't
+            access field names from this context.
+        """
         token = Token(string, metadata)
         return [str(tkn) for tkn in self.run([token])]
 


Test patch:
diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py
index dd17125..20854ce 100644
--- a/tests/test_pipeline.py
+++ b/tests/test_pipeline.py
@@ -194,6 +194,25 @@ class TestRun(BaseTestPipeline):
 
         assert received == ["foo", "FOO"]
 
+    def test_skip_pipeline_function(self) -> None:
+        def upper(t, *args):
+            return t.upper()
+
+        def lower(t, *args):
+            return t.lower()
+
+        self.pipeline.add(upper)
+        self.pipeline.skip(upper, ["no_upper", "nothing"])
+        assert self.pipeline.run(["Foo"]) == ["FOO"]
+
+        self.pipeline.add(lower)
+        self.pipeline.skip(lower, ["no_lower", "nothing"])
+        assert self.pipeline.run(["Foo"]) == ["foo"]
+
+        assert self.pipeline.run(["Foo"], field_name="no_lower") == ["FOO"]
+        assert self.pipeline.run(["Foo"], field_name="no_upper") == ["foo"]
+        assert self.pipeline.run(["Foo"], field_name="nothing") == ["Foo"]
+
 
 class TestSerialize(BaseTestPipeline):
     def test_serialize_returns_array_of_registered_function_labels(self):