Repository: owncloud/ocis
Pull request #7560 (2023-10-23 06:27:54)

Bump postcss-safe-parser from 6.0.0 to 7.0.0 in /services/idp
Bumps [postcss-safe-parser](https://github.com/postcss/postcss-safe-parser) from 6.0.0 to 7.0.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/postcss/postcss-safe-parser/blob/main/CHANGELOG.md">postcss-safe-parser's changelog</a>.</em></p>
<blockquote>
<h1>Change Log</h1>
<p>This project adheres to <a href="http://semver.org/">Semantic Versioning</a>.</p>
<h2>7.0</h2>
<ul>
<li>Removed Node.js 16, 14, and 12 support.</li>
<li>Added <code>Root#source.end</code>.</li>
</ul>
<h2>6.0</h2>
<ul>
<li>Removed Node.js 10 support.</li>
<li>Moved <code>postcss</code> to <code>peerDependencies</code>.</li>
</ul>
<h2>5.0.2</h2>
<ul>
<li>Added funding links.</li>
</ul>
<h2>5.0.1</h2>
<ul>
<li>Fixed parsing missed semicolon.</li>
</ul>
<h2>5.0</h2>
<ul>
<li>Removed support for Node.js 6.x, 8.x, 11.x, and 13.x versions.</li>
<li>Moved to PostCSS 8.0.</li>
</ul>
<h2>4.0.2</h2>
<ul>
<li>Fix parsing <code>:;</code>.</li>
</ul>
<h2>4.0.1</h2>
<ul>
<li>Remove development file from npm package.</li>
</ul>
<h2>4.0</h2>
<ul>
<li>Remove Node.js 9 and Node.js 4 support.</li>
<li>Remove IE and “dead” browsers from Babel.</li>
<li>Use PostCSS 7.0.</li>
</ul>
<h2>3.0.1</h2>
<ul>
<li>Fix parsing IE filter with missed semicolon.</li>
</ul>
<h2>3.0</h2>
<ul>
<li>Use PostCSS 6.0.</li>
<li>Use <code>babel-preset-env</code>.</li>
</ul>
<h2>2.0.1</h2>
<ul>
<li>Do not fall on colon instead of semicolon in the declaration end.</li>
</ul>
<h2>2.0</h2>
<ul>
<li>Use PostCSS 5.2 core tokenizer.</li>
</ul>
<h2>1.0.7</h2>
<ul>
<li>Parse new lines according W3C CSS syntax specification.</li>
</ul>
<h2>1.0.6</h2>
<ul>
<li>Fix package dependencies.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/bdd0c12084dd13d7d485704840d8c06bb884eb22"><code>bdd0c12</code></a> Release 7.0 version</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/2c04213cdf83d6e7c798151ca7b983dcf018108b"><code>2c04213</code></a> Update PostCSS peer req</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/52ebd9cc01cb20bd74cf2ed61b13d6ed54fd50c8"><code>52ebd9c</code></a> Remove Node 12, 14, 16 support</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/cb8fcdcd1fa9a127bc420a5d758c8048c880bb94"><code>cb8fcdc</code></a> Update dependencies and practies</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/abf10f9cf55ac9106a76aebf3264224547e2338c"><code>abf10f9</code></a> Bump postcss from 8.4.21 to 8.4.31 (<a href="https://redirect.github.com/postcss/postcss-safe-parser/issues/38">#38</a>)</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/5e931f4b8d0ccc8a4403de4dfdc405d69e2cab39"><code>5e931f4</code></a> Update dependencies</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/39635379299b14d9c546c3b2fe453c4f7358a152"><code>3963537</code></a> Update CI config</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/e119881ab2133f53f6467b9ad3730c1dfcb0175c"><code>e119881</code></a> Add funding option</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/64e0aecb9e153327936c8d240669497566b5be93"><code>64e0aec</code></a> Add Tidelift links</li>
<li><a href="https://github.com/postcss/postcss-safe-parser/commit/e0089d90a9ba51b0ce628781320c651575740ee9"><code>e0089d9</code></a> Add security note</li>
<li>Additional commits viewable in <a href="https://github.com/postcss/postcss-safe-parser/compare/6.0.0...7.0.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=postcss-safe-parser&package-manager=npm_and_yarn&previous-version=6.0.0&new-version=7.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

update-docs[bot]: Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a [changelog](https://github.com/owncloud/ocis/blob/master/changelog/README.md) item based on your changes.

sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=owncloud_ocis&pullRequest=7560)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=owncloud_ocis&pullRequest=7560&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=owncloud_ocis&pullRequest=7560&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=owncloud_ocis&pullRequest=7560&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=owncloud_ocis&pullRequest=7560&metric=coverage&view=list) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=owncloud_ocis&pullRequest=7560&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=owncloud_ocis&pullRequest=7560&metric=new_duplicated_lines_density&view=list)


rhafer: @dependabot rebase
sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=owncloud_ocis&pullRequest=7560)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=owncloud_ocis&pullRequest=7560&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=owncloud_ocis&pullRequest=7560&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=owncloud_ocis&pullRequest=7560&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=owncloud_ocis&pullRequest=7560&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=owncloud_ocis&pullRequest=7560&metric=coverage&view=list) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=owncloud_ocis&pullRequest=7560&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=owncloud_ocis&pullRequest=7560&metric=new_duplicated_lines_density&view=list)



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 89914649057..5f00a1695b3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -34,6 +34,7 @@ The following sections list the changes for unreleased.
 * Enhancement - Config for disabling Web extensions: [#7486](https://github.com/owncloud/ocis/pull/7486)
 * Enhancement - Add login URL config: [#7317](https://github.com/owncloud/ocis/pull/7317)
 * Enhancement - New value `auto` for NOTIFICATIONS_SMTP_AUTHENTICATION: [#7356](https://github.com/owncloud/ocis/issues/7356)
+* Enhancement - Tika content extraction cleanup for search: [#7553](https://github.com/owncloud/ocis/pull/7553)
 * Enhancement - SSE for messaging: [#6992](https://github.com/owncloud/ocis/pull/6992)
 * Enhancement - Improve SSE format: [#7325](https://github.com/owncloud/ocis/pull/7325)
 * Enhancement - Keyword Query Language (KQL) search syntax: [#7212](https://github.com/owncloud/ocis/pull/7212)
@@ -255,6 +256,21 @@ The following sections list the changes for unreleased.
 
    https://github.com/owncloud/ocis/issues/7356
 
+* Enhancement - Tika content extraction cleanup for search: [#7553](https://github.com/owncloud/ocis/pull/7553)
+
+   So far it has not been possible to determine whether the content for search should be cleaned up
+   of 'stop words' or not. Stop words are filling words like "I, you, have, am" etc and defined by the
+   search engine.
+
+   The behaviour can now be set with the newly introduced settings option
+   `SEARCH_EXTRACTOR_TIKA_CLEAN_STOP_WORDS=false` which is enabled by default.
+
+   In addition, the stop word cleanup is no longer as aggressive and now ignores numbers, urls,
+   basically everything except the defined stop words.
+
+   https://github.com/owncloud/ocis/issues/6674
+   https://github.com/owncloud/ocis/pull/7553
+
 * Enhancement - SSE for messaging: [#6992](https://github.com/owncloud/ocis/pull/6992)
 
    So far, sse has only been used to exchange messages between the server and the client. In order to
diff --git a/changelog/unreleased/enhancement-search-content-extraction-cleanup.md b/changelog/unreleased/enhancement-search-content-extraction-cleanup.md
new file mode 100644
index 00000000000..566b15f7792
--- /dev/null
+++ b/changelog/unreleased/enhancement-search-content-extraction-cleanup.md
@@ -0,0 +1,15 @@
+Enhancement: Tika content extraction cleanup for search
+
+So far it has not been possible to determine whether the
+content for search should be cleaned up of 'stop words' or not.
+Stop words are filling words like "I, you, have, am" etc and
+defined by the search engine.
+
+The behaviour can now be set with the newly introduced settings option `SEARCH_EXTRACTOR_TIKA_CLEAN_STOP_WORDS=false`
+which is enabled by default.
+
+In addition, the stop word cleanup is no longer as aggressive and now ignores numbers, urls,
+basically everything except the defined stop words.
+
+https://github.com/owncloud/ocis/pull/7553
+https://github.com/owncloud/ocis/issues/6674
diff --git a/services/idp/package.json b/services/idp/package.json
index 55774e6c1fa..584e854eb8c 100644
--- a/services/idp/package.json
+++ b/services/idp/package.json
@@ -142,7 +142,7 @@
     "postcss-loader": "4.3.0",
     "postcss-normalize": "10.0.1",
     "postcss-preset-env": "9.1.2",
-    "postcss-safe-parser": "6.0.0",
+    "postcss-safe-parser": "7.0.0",
     "react-dev-utils": "^12.0.1",
     "resolve": "1.22.4",
     "resolve-url-loader": "^5.0.0",
diff --git a/services/idp/pnpm-lock.yaml b/services/idp/pnpm-lock.yaml
index 1f4ec93d2db..e3382799b91 100644
--- a/services/idp/pnpm-lock.yaml
+++ b/services/idp/pnpm-lock.yaml
@@ -215,8 +215,8 @@ devDependencies:
     specifier: 9.1.2
     version: 9.1.2(postcss@8.4.31)
   postcss-safe-parser:
-    specifier: 6.0.0
-    version: 6.0.0(postcss@8.4.31)
+    specifier: 7.0.0
+    version: 7.0.0(postcss@8.4.31)
   react-dev-utils:
     specifier: ^12.0.1
     version: 12.0.1(eslint@7.32.0)(typescript@4.9.5)(webpack@4.47.0)
@@ -9903,11 +9903,11 @@ packages:
       postcss: 8.4.31
     dev: true
 
-  /postcss-safe-parser@6.0.0(postcss@8.4.31):
-    resolution: {integrity: sha512-FARHN8pwH+WiS2OPCxJI8FuRJpTVnn6ZNFiqAM2aeW2LwTHWWmWgIyKC6cUo0L8aeKiF/14MNvnpls6R2PBeMQ==}
-    engines: {node: '>=12.0'}
+  /postcss-safe-parser@7.0.0(postcss@8.4.31):
+    resolution: {integrity: sha512-ovehqRNVCpuFzbXoTb4qLtyzK3xn3t/CUBxOs8LsnQjQrShaB4lKiHoVqY8ANaC0hBMHq5QVWk77rwGklFUDrg==}
+    engines: {node: '>=18.0'}
     peerDependencies:
-      postcss: ^8.3.3
+      postcss: ^8.4.31
     dependencies:
       postcss: 8.4.31
     dev: true
diff --git a/services/search/README.md b/services/search/README.md
index 82df9ca3dc3..739fc3a6771 100644
--- a/services/search/README.md
+++ b/services/search/README.md
@@ -70,6 +70,9 @@ When the search service can reach Tika, it begins to read out the content on dem
 
 Content extraction and handling the extracted content can be very resource intensive. Content extraction is therefore limited to files with a certain file size. The default limit is 20MB and can be configured using the `SEARCH_CONTENT_EXTRACTION_SIZE_LIMIT` variable.
 
+When extracting the content you can specify whether filler words are ignored or not.
+To keep them, the environment variable `SEARCH_EXTRACTOR_TIKA_CLEAN_STOP_WORDS` must be set to false.
+
 When using the Tika container and docker-compose, consider the following:
 
 *   See the [ocis_wopi](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_wopi) example.
diff --git a/services/search/pkg/config/content.go b/services/search/pkg/config/content.go
index 1159bf22c35..df753056a68 100644
--- a/services/search/pkg/config/content.go
+++ b/services/search/pkg/config/content.go
@@ -9,5 +9,6 @@ type Extractor struct {
 
 // ExtractorTika configures the Tika extractor
 type ExtractorTika struct {
-	TikaURL string `yaml:"tika_url" env:"SEARCH_EXTRACTOR_TIKA_TIKA_URL" desc:"URL of the tika server."`
+	TikaURL        string `yaml:"tika_url" env:"SEARCH_EXTRACTOR_TIKA_TIKA_URL" desc:"URL of the tika server."`
+	CleanStopWords bool   `yaml:"clean_stop_words" env:"SEARCH_EXTRACTOR_TIKA_CLEAN_STOP_WORDS" desc:"Defines if stop words should be cleaned or not. See the documentation for more details."`
 }
diff --git a/services/search/pkg/config/defaults/defaultconfig.go b/services/search/pkg/config/defaults/defaultconfig.go
index c7575081c71..3f4cce3e0ab 100644
--- a/services/search/pkg/config/defaults/defaultconfig.go
+++ b/services/search/pkg/config/defaults/defaultconfig.go
@@ -43,7 +43,8 @@ func DefaultConfig() *config.Config {
 			Type:             "basic",
 			CS3AllowInsecure: false,
 			Tika: config.ExtractorTika{
-				TikaURL: "http://127.0.0.1:9998",
+				TikaURL:        "http://127.0.0.1:9998",
+				CleanStopWords: true,
 			},
 		},
 		Events: config.Events{
diff --git a/services/search/pkg/content/content.go b/services/search/pkg/content/content.go
index e58f35dc435..ef334e07064 100644
--- a/services/search/pkg/content/content.go
+++ b/services/search/pkg/content/content.go
@@ -1,5 +1,15 @@
 package content
 
+import (
+	"strings"
+
+	"github.com/bbalet/stopwords"
+)
+
+func init() {
+	stopwords.OverwriteWordSegmenter(`[^ ]+`)
+}
+
 // Document wraps all resource meta fields,
 // it is used as a content extraction result.
 type Document struct {
@@ -11,3 +21,7 @@ type Document struct {
 	MimeType string
 	Tags     []string
 }
+
+func CleanString(content, langCode string) string {
+	return strings.TrimSpace(stopwords.CleanString(content, langCode, true))
+}
diff --git a/services/search/pkg/content/tika.go b/services/search/pkg/content/tika.go
index 1979cdbcdb3..b2953170b48 100644
--- a/services/search/pkg/content/tika.go
+++ b/services/search/pkg/content/tika.go
@@ -5,11 +5,11 @@ import (
 	"fmt"
 	"strings"
 
-	"github.com/bbalet/stopwords"
 	gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
 	provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
 	"github.com/cs3org/reva/v2/pkg/rgrpc/todo/pool"
 	"github.com/google/go-tika/tika"
+
 	"github.com/owncloud/ocis/v2/ocis-pkg/log"
 	"github.com/owncloud/ocis/v2/services/search/pkg/config"
 )
@@ -20,7 +20,8 @@ type Tika struct {
 	*Basic
 	Retriever
 	tika                       *tika.Client
-	contentExtractionSizeLimit uint64
+	ContentExtractionSizeLimit uint64
+	CleanStopWords             bool
 }
 
 // NewTikaExtractor creates a new Tika instance.
@@ -41,7 +42,8 @@ func NewTikaExtractor(gatewaySelector pool.Selectable[gateway.GatewayAPIClient],
 		Basic:                      basic,
 		Retriever:                  newCS3Retriever(gatewaySelector, logger, cfg.Extractor.CS3AllowInsecure),
 		tika:                       tika.NewClient(nil, cfg.Extractor.Tika.TikaURL),
-		contentExtractionSizeLimit: cfg.ContentExtractionSizeLimit,
+		ContentExtractionSizeLimit: cfg.ContentExtractionSizeLimit,
+		CleanStopWords:             cfg.Extractor.Tika.CleanStopWords,
 	}, nil
 }
 
@@ -56,7 +58,7 @@ func (t Tika) Extract(ctx context.Context, ri *provider.ResourceInfo) (Document,
 		return doc, nil
 	}
 
-	if ri.Size > t.contentExtractionSizeLimit {
+	if ri.Size > t.ContentExtractionSizeLimit {
 		t.logger.Info().Interface("ResourceID", ri.Id).Str("Name", ri.Name).Msg("file exceeds content extraction size limit. skipping.")
 		return doc, nil
 	}
@@ -86,8 +88,8 @@ func (t Tika) Extract(ctx context.Context, ri *provider.ResourceInfo) (Document,
 		}
 	}
 
-	if lang, _ := t.tika.LanguageString(ctx, doc.Content); lang != "" {
-		doc.Content = stopwords.CleanString(doc.Content, lang, true)
+	if langCode, _ := t.tika.LanguageString(ctx, doc.Content); langCode != "" && t.CleanStopWords {
+		doc.Content = CleanString(doc.Content, langCode)
 	}
 
 	return doc, nil


Test patch:
diff --git a/services/search/pkg/content/content_test.go b/services/search/pkg/content/content_test.go
new file mode 100644
index 00000000000..17631fe4d79
--- /dev/null
+++ b/services/search/pkg/content/content_test.go
@@ -0,0 +1,36 @@
+package content_test
+
+import (
+	"testing"
+
+	. "github.com/stretchr/testify/assert"
+
+	"github.com/owncloud/ocis/v2/services/search/pkg/content"
+)
+
+func TestCleanContent(t *testing.T) {
+	tests := []struct {
+		given  string
+		expect string
+	}{
+		{
+			given:  "find can keeper should keeper will",
+			expect: "keeper keeper",
+		},
+		{
+			given:  "user1 shares the file to Marie",
+			expect: "user1 shares file marie",
+		},
+		{
+			given:  "content contains https://localhost/remote.php/dav/files/admin/Photos/San%20Francisco.jpg and stop word",
+			expect: "content contains https://localhost/remote.php/dav/files/admin/photos/san%20francisco.jpg stop word",
+		},
+	}
+
+	for _, tc := range tests {
+		tc := tc
+		t.Run(tc.given, func(t *testing.T) {
+			Equal(t, tc.expect, content.CleanString(tc.given, "en"))
+		})
+	}
+}
diff --git a/services/search/pkg/content/tika_test.go b/services/search/pkg/content/tika_test.go
index fa261182417..fbb5c0578b1 100644
--- a/services/search/pkg/content/tika_test.go
+++ b/services/search/pkg/content/tika_test.go
@@ -11,11 +11,12 @@ import (
 	provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
+	"github.com/stretchr/testify/mock"
+
 	"github.com/owncloud/ocis/v2/ocis-pkg/log"
 	conf "github.com/owncloud/ocis/v2/services/search/pkg/config/defaults"
 	"github.com/owncloud/ocis/v2/services/search/pkg/content"
 	contentMocks "github.com/owncloud/ocis/v2/services/search/pkg/content/mocks"
-	"github.com/stretchr/testify/mock"
 )
 
 var _ = Describe("Tika", func() {
@@ -48,6 +49,7 @@ var _ = Describe("Tika", func() {
 
 			cfg := conf.DefaultConfig()
 			cfg.Extractor.Tika.TikaURL = srv.URL
+			cfg.Extractor.Tika.CleanStopWords = true
 
 			var err error
 			tika, err = content.NewTikaExtractor(nil, log.NewLogger(), cfg)
@@ -82,7 +84,7 @@ var _ = Describe("Tika", func() {
 		})
 
 		It("removes stop words", func() {
-			body = "body to test stop words!!! I, you, he, she, it, we, you, they, stay"
+			body = "body to test stop words!!! against almost everyone"
 			language = "en"
 
 			doc, err := tika.Extract(context.TODO(), &provider.ResourceInfo{
@@ -90,7 +92,20 @@ var _ = Describe("Tika", func() {
 				Size: 1,
 			})
 			Expect(err).ToNot(HaveOccurred())
-			Expect(doc.Content).To(Equal("body test stop words i stay "))
+			Expect(doc.Content).To(Equal("body test stop words!!!"))
+		})
+
+		It("keeps stop words", func() {
+			body = "body to test stop words!!! against almost everyone"
+			language = "en"
+
+			tika.CleanStopWords = false
+			doc, err := tika.Extract(context.TODO(), &provider.ResourceInfo{
+				Type: provider.ResourceType_RESOURCE_TYPE_FILE,
+				Size: 1,
+			})
+			Expect(err).ToNot(HaveOccurred())
+			Expect(doc.Content).To(Equal(body))
 		})
 	})
 })