Repository: semgrep/semgrep (language: python)

Issue:
Add Rust parsing support
Adds Rust parsing support. This is up to the CST -> AST level. There is no pattern support yet.

Related PRs:

https://github.com/returntocorp/ocaml-tree-sitter/pull/134
https://github.com/returntocorp/ocaml-tree-sitter-lang/pull/5
https://github.com/returntocorp/pfff/pull/370
https://github.com/returntocorp/semgrep-grammars/pull/14

At the moment only one test on the Semgrep side is failing. It's weird since it parses correctly from tree-sitter.

Also, because there were so many language constructs that needed to be added, there are still a lot of TODOs for parts like flow control. I'm not entirely sure what those parts of the code are supposed to do yet. Some parts of the tree-sitter parser still need fixing also. For example, the `use` statement is an expression when it comes out of tree-sitter, but the Rust language docs say you can simplify it to a dotted identifier.
suggest a config like --config=p/r2c-ci if no config provided
Related to @underyx issue about supporting -c for config  https://github.com/returntocorp/semgrep/issues/2112 , I just spent a full minute trying to remember the name for our "default" rule pack. Let's make that easier to find on the CLI, by suggest a config like --config=p/r2c-ci if no config provided, in --help, or if the current config is empty
PHP Class in pattern results in parsing error
**Describe the bug**
When using `class $Class` or `class SomeThing` in a pattern, it results in an invalid pattern error

**To Reproduce**
https://semgrep.dev/s/gLpR

**Expected behavior**
There should be a match

**What is the priority of the bug to you?**
P1

**Environment**
semgrep.dev and semgrep cli 0.35.0 on MacOSX (installed via brew)

**Notes**
- Using a pattern like `pattern: public function __destruct() { ... }` also produces a parsing error. Seems like to be related to the `public` keyword
- There is an extremely weird behaviour when trying to match functions/methods, see https://semgrep.dev/s/Q5wJ which won't have any match, but then adding `...` before any `function __` (ie `... function __toString() { ... }`) will result in one match



Discussion / hints:
As suggested, I'll break up this PR into multiple parts instead. The first PR is here: https://github.com/returntocorp/semgrep/pull/2365

https://semgrep.dev/s/gLpR?version=develop works! This fix will go out in the next release.
Also works with the new release: https://semgrep.dev/s/gLpR?version=0.38.0


Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 451f859b93bb..a14d96d2bf0a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,32 @@
 
 <!-- insertion point -->
 
+## [1.91.0](https://github.com/returntocorp/semgrep/releases/tag/v1.91.0) - 2024-10-10
+
+
+### Added
+
+
+- Type inference in the Pro engine has been improved for class fields in
+  TypeScript that are assigned a new instance but lack an explicit type
+  definition. When no explicit type is provided for a class field, its type is
+  inferred from the type of the expression assigned to it. For example, in the
+  class definition `class Foo { private readonly bar = new Bar(); }`, the type of
+  `bar` is inferred to be `Bar`. (code-7635)
+- Cargo.lock parser can now associate dependencies with lockfile line numbers (sc-1140)
+
+
+### Fixed
+
+
+- Address python `rich.errors.LiveError` where attempting to display multiple progress bars
+  raises an exception as flagged in #10562. (grow-414)
+- C: Fix a regression causing pattern `-n` to sometimes not match code `-n`. (saf-1592)
+- When a scan runs into an exception, the app is appropriately notified
+  about the failure. Previously, in the app, it would seem to the user
+  that the scan is still in progress. (sms-502)
+
+
 ## [1.90.0](https://github.com/returntocorp/semgrep/releases/tag/v1.90.0) - 2024-09-25
 
 
diff --git a/Dockerfile b/Dockerfile
index 5c8574b127b0..752ba936be6c 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -109,6 +109,7 @@ RUN apk add --no-cache bash build-base git make opam
 # - https://github.com/Homebrew/homebrew-core/blob/master/Formula/semgrep.rb
 #TODO: switch to 5.2.0 at some point
 RUN opam init --disable-sandboxing -v && opam switch create 4.14.0 -v
+RUN opam update
 
 # Install semgrep-core build dependencies
 WORKDIR /src/semgrep
diff --git a/Makefile b/Makefile
index d120162afbdc..1f0e47b82114 100644
--- a/Makefile
+++ b/Makefile
@@ -237,7 +237,18 @@ core-test-e2e:
 # path, then recent versions of opam crash with a 'git ls-files fatal error'
 # about some 'libs/ocaml-tree-sitter-core/../../.git/...' not being a git
 # repo.
-REQUIRED_DEPS = ./ ./libs/ocaml-tree-sitter-core/tree-sitter.opam ./dev/required.opam
+#
+# EXTRA_OPAM_DEPS allows us to add more opam files when building semgrep
+# as part of a larger project (e.g. semgrep-proprietary). Using a single
+# 'opam install' command to install all the dependencies allows us to detect
+# version constraints incompatibilities.
+#
+REQUIRED_DEPS = \
+ ./ \
+ ./libs/ocaml-tree-sitter-core/tree-sitter.opam \
+  ./dev/required.opam \
+  $(EXTRA_OPAM_DEPS)
+
 OPTIONAL_DEPS = $(REQUIRED_DEPS) ./dev/optional.opam
 
 # This target is portable; it only assumes you have 'gcc', 'opam' and
diff --git a/cli/setup.py b/cli/setup.py
index cf8e0e402041..05e3d15225c2 100644
--- a/cli/setup.py
+++ b/cli/setup.py
@@ -139,7 +139,7 @@ def find_executable(env_name, exec_name):
 
 setuptools.setup(
     name="semgrep",
-    version="1.90.0",
+    version="1.91.0",
     author="Semgrep Inc.",
     author_email="support@semgrep.com",
     description="Lightweight static analysis for many languages. Find bug variants with patterns that look like source code.",
diff --git a/cli/src/semdep/parsers/cargo.py b/cli/src/semdep/parsers/cargo.py
index 965d32cd26bc..391742cfd4c0 100644
--- a/cli/src/semdep/parsers/cargo.py
+++ b/cli/src/semdep/parsers/cargo.py
@@ -1,5 +1,6 @@
 from pathlib import Path
 from typing import Generator
+from typing import List
 from typing import Optional
 
 from semgrep.semgrep_interfaces.semgrep_output_v1 import Cargo
@@ -13,7 +14,7 @@
 def parse_cargo(
     lockfile_path: Path, lockfile_text: str, manifest_text: Optional[str]
 ) -> Generator[FoundDependency, None, None]:
-    def parse_dep(s: str) -> FoundDependency:
+    def parse_dep(s: str, raw_lines: List[str]) -> FoundDependency:
         lines = s.split("\n")[1:]
         dep = lines[0].split("=")[1].strip()[1:-1]
         version = lines[1].split("=")[1].strip()[1:-1]
@@ -29,7 +30,9 @@ def parse_dep(s: str) -> FoundDependency:
             allowed_hashes=hash,
             transitivity=Transitivity(Unknown()),
             lockfile_path=Fpath(str(lockfile_path)),
+            line_number=raw_lines.index(lines[0]) + 1,
         )
 
+    raw_lines = lockfile_text.split("\n")
     deps = lockfile_text.split("[[package]]")[1:]
-    yield from (parse_dep(dep) for dep in deps)
+    yield from (parse_dep(dep, raw_lines) for dep in deps)
diff --git a/cli/src/semgrep/__init__.py b/cli/src/semgrep/__init__.py
index 982ff42cdadc..f1d3d66feb92 100644
--- a/cli/src/semgrep/__init__.py
+++ b/cli/src/semgrep/__init__.py
@@ -1,1 +1,1 @@
-__VERSION__ = "1.90.0"
+__VERSION__ = "1.91.0"
diff --git a/cli/src/semgrep/app/scans.py b/cli/src/semgrep/app/scans.py
index 06865a6d68ac..a45352f77890 100644
--- a/cli/src/semgrep/app/scans.py
+++ b/cli/src/semgrep/app/scans.py
@@ -484,15 +484,16 @@ def report_findings(
         # minutes to wait for completion. Eventually, this wait may
         # be configurable as we see larger scans and increased backend
         # load.
-        try_until = datetime.utcnow() + timedelta(minutes=30)
-        slow_down_after = datetime.utcnow() + timedelta(minutes=2)
+        now = datetime.now().replace(tzinfo=None)
+        try_until = now + timedelta(minutes=30)
+        slow_down_after = now + timedelta(minutes=2)
 
         while True:
             # old: was also logging {json.dumps(complete.to_json(), indent=4)}
             # alt: save it in ~/.semgrep/logs/complete.json?
             logger.debug(f"Sending /complete")
 
-            if datetime.utcnow() > try_until:
+            if datetime.now().replace(tzinfo=None) > try_until:
                 # let the backend know we won't be trying again
                 complete.final_attempt = True
 
@@ -522,4 +523,4 @@ def report_findings(
                 )
 
             progress_bar.advance(complete_task)
-            sleep(5 if datetime.utcnow() < slow_down_after else 30)
+            sleep(5 if datetime.now().replace(tzinfo=None) < slow_down_after else 30)
diff --git a/cli/src/semgrep/commands/ci.py b/cli/src/semgrep/commands/ci.py
index 19c85c03c07f..445f51bd0bbd 100644
--- a/cli/src/semgrep/commands/ci.py
+++ b/cli/src/semgrep/commands/ci.py
@@ -180,6 +180,13 @@ def fix_head_if_github_action(metadata: GitMeta) -> None:
     is_flag=True,
     hidden=True,
 )
+@click.option(
+    "--x-dump-rule-partitions",
+    "dump_n_rule_partitions",
+    type=int,
+    default=0,
+    hidden=True,
+)
 @handle_command_errors
 def ci(
     ctx: click.Context,
@@ -241,6 +248,7 @@ def ci(
     verbose: bool,
     path_sensitive: bool,
     enable_experimental_requirements: bool,
+    dump_n_rule_partitions: Optional[int],
 ) -> None:
     state = get_state()
 
@@ -292,6 +300,10 @@ def ci(
             )
             sys.exit(FATAL_EXIT_CODE)
         elif token:
+            # Dumping partitions implies a dry run because we are not
+            # scanning or uploading results to the app.
+            if dump_n_rule_partitions:
+                dry_run = True
             scan_handler = ScanHandler(dry_run=dry_run)
         else:  # impossible state… until we break the code above
             raise RuntimeError("The token and/or config are misconfigured")
@@ -546,15 +558,22 @@ def ci(
             "diff_depth": diff_depth,
             "capture_core_stderr": capture_core_stderr,
             "enable_experimental_requirements": enable_experimental_requirements,
+            "dump_n_rule_partitions": dump_n_rule_partitions,
         }
 
         try:
             start = time.time()
 
-            if scan_handler and not scan_handler.enabled_products:
-                raise SemgrepError(
-                    "No products are enabled for this organization. Please enable a product in the Settings > Deployment tab of Semgrep Cloud Platform or reach out to support@semgrep.com for assistance."
-                )
+            if scan_handler:
+                if not scan_handler.enabled_products:
+                    raise SemgrepError(
+                        "No products are enabled for this organization. Please enable a product in the Settings > Deployment tab of Semgrep Cloud Platform or reach out to support@semgrep.com for assistance."
+                    )
+
+                if dump_n_rule_partitions and scan_handler.enabled_products != ["sast"]:
+                    raise SemgrepError(
+                        "--x-dump-rule-partitions is only compatible with SAST."
+                    )
 
             # TODO? we're not passing time_flag below (or matching_explanations),
             # is it indended?
@@ -573,15 +592,23 @@ def ci(
                 _missed_rule_count,
             ) = semgrep.run_scan.run_scan(**run_scan_args)
         except SemgrepError as e:
-            output_handler.handle_semgrep_errors([e])
-            output_handler.output({}, all_targets=set(), filtered_rules=[])
-            logger.info(f"Encountered error when running rules: {e}")
+            # We place output_handler calls after scan_handler calls
+            # because the output handler may raise an exception further
+            # for the top-level error handler to handle.
+            #
+            # We'd still like scan_handler to notify the app for failures
+            # so we do it before calling output handler.
             if isinstance(e, SemgrepError):
                 exit_code = e.code
             else:
                 exit_code = FATAL_EXIT_CODE
             if scan_handler:
                 scan_handler.report_failure(exit_code)
+
+            output_handler.handle_semgrep_errors([e])
+            output_handler.output({}, all_targets=set(), filtered_rules=[])
+            logger.info(f"Encountered error when running rules: {e}")
+
             sys.exit(exit_code)
 
         # Run a separate scan for historical. This is due to the split in how the
diff --git a/cli/src/semgrep/commands/publish.py b/cli/src/semgrep/commands/publish.py
index 3560103e3f83..9a8ed36769fd 100644
--- a/cli/src/semgrep/commands/publish.py
+++ b/cli/src/semgrep/commands/publish.py
@@ -13,7 +13,6 @@
 from semgrep.commands.wrapper import handle_command_errors
 from semgrep.config_resolver import get_config
 from semgrep.error import FATAL_EXIT_CODE
-from semgrep.git import clean_project_url
 from semgrep.git import get_project_url
 from semgrep.state import get_state
 from semgrep.test import get_config_filenames
@@ -176,9 +175,7 @@ def _upload_rule(
     rule = rules[0]
 
     # add metadata about the origin of the rule
-    origin_note = (
-        f"published from {rule_file} in {clean_project_url(get_project_url() or '')}"
-    )
+    origin_note = f"published from {rule_file} in {get_project_url()}"
     rule.metadata["rule-origin-note"] = origin_note
 
     request_json = {
diff --git a/cli/src/semgrep/config_resolver.py b/cli/src/semgrep/config_resolver.py
index 733542946af4..5c01e8cb1677 100644
--- a/cli/src/semgrep/config_resolver.py
+++ b/cli/src/semgrep/config_resolver.py
@@ -21,14 +21,12 @@
 from urllib.parse import urlsplit
 
 import requests
-from rich import progress
 from ruamel.yaml import YAMLError
 
 import semgrep.semgrep_interfaces.semgrep_output_v1 as out
 from semgrep import __VERSION__
 from semgrep import tracing
 from semgrep.app import auth
-from semgrep.console import console
 from semgrep.constants import CLI_RULE_ID
 from semgrep.constants import Colors
 from semgrep.constants import DEFAULT_SEMGREP_APP_CONFIG_URL
@@ -433,14 +431,7 @@ def parse_config_files(
     """
     config = {}
     errors: List[SemgrepError] = []
-    for config_id, contents, config_path in progress.track(
-        loaded_config_infos,
-        description=f"  parsing {len(loaded_config_infos)} rules",
-        transient=True,
-        # expected to take just 2-3 seconds with less than 500
-        disable=len(loaded_config_infos) < 500,
-        console=console,
-    ):
+    for config_id, contents, config_path in loaded_config_infos:
         try:
             if not config_id:  # registry rules don't have config ids
                 # Note: we must disambiguate registry sourced remote rules from
diff --git a/cli/src/semgrep/git.py b/cli/src/semgrep/git.py
index 6e54e9c53050..d5c5f896eeb5 100644
--- a/cli/src/semgrep/git.py
+++ b/cli/src/semgrep/git.py
@@ -65,6 +65,8 @@ def git_check_output(command: Sequence[str], cwd: Optional[str] = None) -> str:
 
                 - the git binary is not available
                 - the current working directory is not a git repository
+                - the baseline commit is not a parent of the current commit
+                    (if you are running through semgrep-app, check if you are setting `SEMGREP_BRANCH` or `SEMGREP_BASELINE_COMMIT` properly)
                 - the current working directory is not marked as safe
                     (fix with `git config --global --add safe.directory $(pwd)`)
 
@@ -76,19 +78,23 @@ def git_check_output(command: Sequence[str], cwd: Optional[str] = None) -> str:
 
 def get_project_url() -> Optional[str]:
     """
-    Returns the current git project's default remote URL, or None if not a git project / no remote
+    Returns the current git project's default remote URL, or None if not a git project / no remote.
+    NOTE: We need to ensure that we clean the URL to remove any credentials as Gitlab includes
+    a token in the URL (e.g.`https://gitlab-ci-token):${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>`)
+    which is sensitive information that we should not expose.
     """
+    project_url = None
     try:
-        remote_url = git_check_output(["git", "ls-remote", "--get-url"])
-        return clean_project_url(remote_url)
+        project_url = git_check_output(["git", "ls-remote", "--get-url"])
     except Exception as e:
         logger.debug(f"Failed to get project url from 'git ls-remote': {e}")
         try:
             # add \n to match urls from git ls-remote (backwards compatibility)
-            return manually_search_file(".git/config", ".com", "\n")
+            project_url = manually_search_file(".git/config", ".com", "\n")
         except Exception as e:
             logger.debug(f"Failed to get project url from .git/config: {e}")
             return None
+    return clean_project_url(project_url) if project_url else None
 
 
 def clean_project_url(url: str) -> str:
diff --git a/cli/src/semgrep/rpc_call.py b/cli/src/semgrep/rpc_call.py
index fa19ea9c7d1b..527ca4d3dd83 100644
--- a/cli/src/semgrep/rpc_call.py
+++ b/cli/src/semgrep/rpc_call.py
@@ -68,5 +68,13 @@ def resolve_dependencies(
     )
     if ret is None:
         logger.warning("failed to resolve dependencies")
-        return None
+    return None
+
+
+def dump_rule_partitions(args: out.DumpRulePartitionsParams) -> bool:
+    call = out.FunctionCall(out.CallDumpRulePartitions(args))
+    ret: Optional[out.RetDumpRulePartitions] = rpc_call(call, out.RetDumpRulePartitions)
+    if ret is None:
+        logger.error("Failed to dump rule partitions")
+        return out.RetDumpRulePartitions(False).value
     return ret.value
diff --git a/cli/src/semgrep/run_scan.py b/cli/src/semgrep/run_scan.py
index c686cb4795dd..3502c56a7715 100644
--- a/cli/src/semgrep/run_scan.py
+++ b/cli/src/semgrep/run_scan.py
@@ -68,6 +68,7 @@
 from semgrep.output import OutputSettings
 from semgrep.output_extra import OutputExtra
 from semgrep.profile_manager import ProfileManager
+from semgrep.rpc_call import dump_rule_partitions
 from semgrep.rule import Rule
 from semgrep.rule_match import RuleMatches
 from semgrep.rule_match import RuleMatchMap
@@ -423,6 +424,7 @@ def run_scan(
     path_sensitive: bool = False,
     capture_core_stderr: bool = True,
     enable_experimental_requirements: bool = False,
+    dump_n_rule_partitions: Optional[int] = None,
 ) -> Tuple[
     RuleMatchMap,
     List[SemgrepError],
@@ -533,6 +535,19 @@ def run_scan(
         ]
     filtered_rules = filter_exclude_rule(filtered_rules, exclude_rule)
 
+    if dump_n_rule_partitions:
+        rules = {"rules": [r.raw for r in filtered_rules]}
+        output_dir = get_state().env.user_data_folder
+        args = out.DumpRulePartitionsParams(
+            out.RawJson(rules), dump_n_rule_partitions, out.Fpath(str(output_dir))
+        )
+        ok = dump_rule_partitions(args)
+        if not ok:
+            logger.error("An error occurred while dumping rule partitions.")
+            sys.exit(2)
+        logger.info(f"Successfully dumped rule partitions to {output_dir}")
+        sys.exit(0)
+
     output_handler.handle_semgrep_errors(config_errors)
     real_config_errors = select_real_errors(config_errors)
     if not pattern:
diff --git a/cli/src/semgrep/semgrep_interfaces b/cli/src/semgrep/semgrep_interfaces
index bc038442cb9a..2351c5e528cb 160000
--- a/cli/src/semgrep/semgrep_interfaces
+++ b/cli/src/semgrep/semgrep_interfaces
@@ -1,1 +1,1 @@
-Subproject commit bc038442cb9abe581b78f2190d2d8ec8a25f65f4
+Subproject commit 2351c5e528cb7430422208dc66707894c066b508
diff --git a/dune-project b/dune-project
index 3e03f5c0ff3b..da79d1df51a8 100644
--- a/dune-project
+++ b/dune-project
@@ -21,7 +21,7 @@
 (generate_opam_files)
 
 ;; set here so the semgrep package below can use it and we can easily bump it
-(version 1.90.0)
+(version 1.91.0)
 
 ;; Default attributes of opam packages
 (source (github semgrep/semgrep))
@@ -77,6 +77,7 @@ the other programming languages supported by atdgen."
     (lib_parsing (>= 1.5.5))
     (profiling (>= 1.5.5))
     (atdgen (>= 2.8.0))
+    (uuidm (>= 0.9.9))
   )
 )
 
@@ -515,7 +516,7 @@ For more information see https://semgrep.dev
     (conf-libcurl (= 1)) ; force older version of conf-libcurl to make windows work
     ; web stuff
     uri
-    uuidm
+    (uuidm (>= 0.9.9))
     ; cohttp >= 6.0.0 requires opam 2.1.0 which used to not available in Windows
     ; TODO: now that opam 2.2.0 is out and support windows, upgrade to 6.0.0
     (cohttp (= 5.3.0))
diff --git a/flake.lock b/flake.lock
index a2aab2ce7136..c83d08ba1422 100644
--- a/flake.lock
+++ b/flake.lock
@@ -21,11 +21,11 @@
         "systems": "systems"
       },
       "locked": {
-        "lastModified": 1710146030,
-        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
+        "lastModified": 1726560853,
+        "narHash": "sha256-X6rJYSESBVr3hBoH0WbKE5KvhPU5bloyZ2L4K60/fPQ=",
         "owner": "numtide",
         "repo": "flake-utils",
-        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
+        "rev": "c1dfcf08411b08f6b8615f7d8971a2bfa81d5e8a",
         "type": "github"
       },
       "original": {
@@ -52,11 +52,11 @@
     },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1718530797,
-        "narHash": "sha256-pup6cYwtgvzDpvpSCFh1TEUjw2zkNpk8iolbKnyFmmU=",
+        "lastModified": 1727634051,
+        "narHash": "sha256-S5kVU7U82LfpEukbn/ihcyNt2+EvG7Z5unsKW9H/yFA=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "b60ebf54c15553b393d144357375ea956f89e9a9",
+        "rev": "06cf0e1da4208d3766d898b7fdab6513366d45b9",
         "type": "github"
       },
       "original": {
@@ -81,11 +81,11 @@
         "opam2json": "opam2json"
       },
       "locked": {
-        "lastModified": 1712645768,
-        "narHash": "sha256-9dUh8nElGtC74Q4gIDV6DM0FKgF1oXh0PUkCxdbp+sg=",
+        "lastModified": 1716292162,
+        "narHash": "sha256-UOJNCbqvxABD56JZtZkv3C9ufdqrs7/Ep4AKkCHgPuo=",
         "owner": "tweag",
         "repo": "opam-nix",
-        "rev": "464863fba44c7ecc50bd1a2967274482a2c33daf",
+        "rev": "1d3cbd6d3f247db77cb581c88c9a1d72e4acad60",
         "type": "github"
       },
       "original": {
@@ -129,11 +129,11 @@
     "opam-repository_2": {
       "flake": false,
       "locked": {
-        "lastModified": 1720795368,
-        "narHash": "sha256-sFOKkWOq00oqrS7BJWIdLOYeMJbo05SmRrFRLKug0PI=",
+        "lastModified": 1727704097,
+        "narHash": "sha256-fBOT/Dl+w6K+7UMkwLeYIcsNuzCIo+q6/X/S1TaHzR8=",
         "owner": "ocaml",
         "repo": "opam-repository",
-        "rev": "80f59f9d12c0e9dda29eb80811ac7689a7ec392b",
+        "rev": "1b4da5019e5ea60af76c94aacc672a7e9659a832",
         "type": "github"
       },
       "original": {
diff --git a/flake.nix b/flake.nix
index 6e2f9efd5327..c244fb44dfde 100644
--- a/flake.nix
+++ b/flake.nix
@@ -221,9 +221,9 @@
 # - [Nix PhD Thesis](https://edolstra.github.io/pubs/phd-thesis.pdf) - Nix
 #   creator's PhD thesis on Nix. ~275 pages but really approachable
 
-
 {
-  description = "Semgrep OSS is a fast, open-source, static analysis tool for searching code, finding bugs, and enforcing code standards at editor, commit, and CI time.";
+  description =
+    "Semgrep OSS is a fast, open-source, static analysis tool for searching code, finding bugs, and enforcing code standards at editor, commit, and CI time.";
   inputs = {
     nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
     # all the follows here are so we don't use diff versions of
@@ -241,10 +241,8 @@
     };
   };
   outputs = { self, nixpkgs, flake-utils, opam-nix, opam-repository }@inputs:
-    let
-      package = "semgrep";
-    in
-    flake-utils.lib.eachDefaultSystem (system:
+    let package = "semgrep";
+    in flake-utils.lib.eachDefaultSystem (system:
       let
         # TODO Use pkgsStatic if on linux
         pkgs = nixpkgs.legacyPackages.${system};
@@ -253,16 +251,12 @@
         opamRepos = [ "${opam-repository}" ];
         lib = pkgs.lib;
         isDarwin = lib.strings.hasSuffix "darwin" system;
-        hasSubmodules = ! builtins.hasAttr "submodules" self || self.submodules;
+        hasSubmodules = !builtins.hasAttr "submodules" self || self.submodules;
         # TODO split out osemgrep and pysemgrep into diff nix files
-      in
-      let
+      in let
 
         # osemgrep/semgrep-core inputs
-        osemgrepInputs = with pkgs; [
-          pcre2
-          tree-sitter
-        ];
+        osemgrepInputs = with pkgs; [ pcre2 tree-sitter ];
         devOpamPackagesQuery = {
           # You can add "development" ocaml packages here. They will get added
           # to the devShell automatically.
@@ -291,14 +285,22 @@
 
         # repos = opamRepos to force newest version of opam
         # pkgs = pkgs to force newest version of nixpkgs instead of using opam-nix's
-        scope = on.buildOpamProject' { pkgs = pkgs; repos = opamRepos; } ./. opamQuery;
+        scope = on.buildOpamProject' {
+          pkgs = pkgs;
+          repos = opamRepos;
+        } ./. opamQuery;
         scopeOverlay = final: prev: {
           # You can add overrides here
           ${package} = prev.${package}.overrideAttrs (prev: {
             # Prevent the ocaml dependencies from leaking into dependent environments
             doNixSupport = false;
-            buildInputs = prev.buildInputs ++ [ final.tsort final.notty final.tyxml final.git-unix ];
-            nativeCheckInputs = [ final.junit_alcotest ];
+            buildInputs = prev.buildInputs ++ [
+              final.tsort
+              final.notty
+              final.tyxml
+              final.git-unix
+              final.junit_alcotest
+            ];
           });
         };
         scope' = scope.overrideScope' scopeOverlay;
@@ -347,11 +349,11 @@
           '';
           # make sure we have submodules
           # See https://github.com/NixOS/nix/pull/7862
-          buildPhase = if hasSubmodules then osemgrep.buildPhase' else osemgrep.buildPhaseFail;
-          nativeCheckInputs = prev.nativeCheckInputs ++ (with pkgs; [
-            cacert
-            git
-          ]);
+          buildPhase = if hasSubmodules then
+            osemgrep.buildPhase'
+          else
+            osemgrep.buildPhaseFail;
+          nativeCheckInputs = (with pkgs; [ cacert git ]);
           # git init is needed so tests work successfully since many rely on git root existing
           checkPhase = ''
             git init
@@ -385,45 +387,45 @@
         # pysemgrep
         #
 
-        pysemgrep = with pythonPackages; buildPythonApplication {
-          # thanks to @06kellyjac
-          pname = "semgrep";
-          inherit (osemgrep) version;
-          src = ./cli;
-          # TODO checks
-          doCheck = false;
-
-          # coupling: anything added to the pysemgrep setup.py should be added here
-          propagatedBuildInputs = [
-            attrs
-            boltons
-            colorama
-            click
-            click-option-group
-            glom
-            requests
-            rich
-            ruamel-yaml
-            tqdm
-            packaging
-            jsonschema
-            wcmatch
-            peewee
-            defusedxml
-            urllib3
-            typing-extensions
-            tomli
-          ];
-          # doesn't work for some reason
-          dontUseSetuptoolsShellHook = true;
-
-          preFixup = ''
-            makeWrapperArgs+=(--prefix PATH : ${osemgrep}/bin)
-          '';
-        };
+        pysemgrep = with pythonPackages;
+          buildPythonApplication {
+            # thanks to @06kellyjac
+            pname = "semgrep";
+            inherit (osemgrep) version;
+            src = ./cli;
+            # TODO checks
+            doCheck = false;
+
+            # coupling: anything added to the pysemgrep setup.py should be added here
+            propagatedBuildInputs = [
+              attrs
+              boltons
+              colorama
+              click
+              click-option-group
+              glom
+              requests
+              rich
+              ruamel-yaml
+              tqdm
+              packaging
+              jsonschema
+              wcmatch
+              peewee
+              defusedxml
+              urllib3
+              typing-extensions
+              tomli
+            ];
+            # doesn't work for some reason
+            dontUseSetuptoolsShellHook = true;
+
+            preFixup = ''
+              makeWrapperArgs+=(--prefix PATH : ${osemgrep}/bin)
+            '';
+          };
         # TODO semgrep-js
-      in
-      {
+      in {
         # For a lot of nix commands, nix uses the cwd's flake. So
         #   nix develop
         # will run the current flake. But because semgrep has submodules we have
diff --git a/languages/lisp/tree-sitter/Parse_clojure_tree_sitter.ml b/languages/lisp/tree-sitter/Parse_clojure_tree_sitter.ml
index f7a09f4814cf..2bddb1ddfa90 100644
--- a/languages/lisp/tree-sitter/Parse_clojure_tree_sitter.ml
+++ b/languages/lisp/tree-sitter/Parse_clojure_tree_sitter.ml
@@ -319,7 +319,7 @@ let parse file =
     (fun cst _extras ->
       let env = { H.file; conv = H.line_col_to_pos file; extra = () } in
       let x = map_source env cst in
-      [ x |> G.raw |> G.exprstmt ])
+      [ x |> G.stmt_of_raw ])
 
 let parse_pattern str =
   H.wrap_parser
diff --git a/languages/protobuf/generic/Parse_protobuf_tree_sitter.ml b/languages/protobuf/generic/Parse_protobuf_tree_sitter.ml
index 678d6a5b3700..a6083677b7d5 100644
--- a/languages/protobuf/generic/Parse_protobuf_tree_sitter.ml
+++ b/languages/protobuf/generic/Parse_protobuf_tree_sitter.ml
@@ -687,8 +687,7 @@ let parse file =
     (fun cst _extras ->
       let env = { H.file; conv = H.line_col_to_pos file; extra = () } in
       let raw = map_source_file env cst in
-      let e = G.RawExpr raw |> G.e in
-      let st = G.exprstmt e in
+      let st = G.stmt_of_raw raw in
       [ st ])
 
 (* todo: special mode to convert Ellipsis in the right construct! *)
diff --git a/libs/ast_generic/AST_generic.ml b/libs/ast_generic/AST_generic.ml
index a4ceb8c70191..458854ebdfff 100644
--- a/libs/ast_generic/AST_generic.ml
+++ b/libs/ast_generic/AST_generic.ml
@@ -839,9 +839,8 @@ and expr_kind =
    * StmtExpr above to wrap stmt if it's not an expr but a stmt
    *)
   | OtherExpr of todo_kind * any list
-  (* experimental alternative to OtherExpr. This allows us to have
-     proper exprs, stmts, etc. embedded in constructs that were not
-     fully translated into the generic AST. *)
+  (* Embed an untranslated or partially translated subtree generated
+     by ocaml-tree-sitter *)
   | RawExpr of raw_tree
 
 and literal =
@@ -1277,6 +1276,9 @@ and stmt_kind =
    * of relying that the any list contains at least one token
    *)
   | OtherStmt of other_stmt_operator * any list
+  (* Embed an untranslated or partially translated subtree generated
+     by ocaml-tree-sitter *)
+  | RawStmt of raw_tree
 
 (* TODO: can also introduce var in some languages
  * less: factorize with for_var_or_expr
@@ -2342,7 +2344,13 @@ let interpolated (lquote, xs, rquote) =
 let keyval k _tarrow v =
   Container (Tuple, Tok.unsafe_fake_bracket [ k; v ]) |> e
 
-let raw x = RawExpr x |> e
+(* Embed untranslated (raw) nodes into the AST *)
+let expr_of_raw (x : any Raw_tree.t) : expr = RawExpr x |> e
+let stmt_of_raw (x : any Raw_tree.t) : stmt = RawStmt x |> s
+
+(* Embed proper AST nodes into an untranslated subtree *)
+let raw_of_expr (x : expr) : any Raw_tree.t = Any (E x)
+let raw_of_stmt (x : stmt) : any Raw_tree.t = Any (S x)
 
 (* ------------------------------------------------------------------------- *)
 (* Parameters *)
diff --git a/libs/ast_generic/AST_generic_to_v1.ml b/libs/ast_generic/AST_generic_to_v1.ml
index b8d617756c24..2b975cda13ea 100644
--- a/libs/ast_generic/AST_generic_to_v1.ml
+++ b/libs/ast_generic/AST_generic_to_v1.ml
@@ -812,6 +812,9 @@ and map_stmt x : B.stmt =
     | OtherStmt (v1, v2) ->
         let v1 = map_other_stmt_operator v1 and v2 = map_of_list map_any v2 in
         `OtherStmt ((v1, map_tok fk), v2)
+    | RawStmt x ->
+        let first_tok, _last_tok = Raw_tree.unsafe_loc x in
+        `OtherStmt (("RawStmt", map_tok first_tok), [])
   in
   skind
 
diff --git a/libs/ast_generic/Meta_AST.ml b/libs/ast_generic/Meta_AST.ml
index 9497eb58fd6b..1628581cf500 100644
--- a/libs/ast_generic/Meta_AST.ml
+++ b/libs/ast_generic/Meta_AST.ml
@@ -894,6 +894,9 @@ and vof_stmt st =
   | OtherStmt (v1, v2) ->
       let v1 = vof_other_stmt_operator v1 and v2 = OCaml.vof_list vof_any v2 in
       OCaml.VSum ("OtherStmt", [ v1; v2 ])
+  | RawStmt x ->
+      let x = vof_raw_tree x in
+      OCaml.VSum ("RawStmt", [ x ])
 
 and vof_condition = function
   | Cond v1 ->
diff --git a/libs/git_wrapper/Git_wrapper.ml b/libs/git_wrapper/Git_wrapper.ml
index 151864b20436..1cb41f3cde45 100644
--- a/libs/git_wrapper/Git_wrapper.ml
+++ b/libs/git_wrapper/Git_wrapper.ml
@@ -280,6 +280,8 @@ let command (_caps_exec : < Cap.exec >) (args : Cmd.args) : string =
 Failed to run %s. Possible reasons:
 - the git binary is not available
 - the current working directory is not a git repository
+- the baseline commit is not a parent of the current commit
+  (if you are running through semgrep-app, check if you are setting `SEMGREP_BRANCH` or `SEMGREP_BASELINE_COMMIT` properly)
 - the current working directory is not marked as safe
   (fix with `git config --global --add safe.directory $(pwd)`)
 
@@ -501,7 +503,8 @@ let run_with_worktree (caps : < Cap.chdir ; Cap.tmp >) ~commit ?branch f =
     | None -> raise (Error "")
   in
   let rand_dir () =
-    let uuid = Uuidm.v `V4 in
+    let rand = Stdlib.Random.State.make_self_init () in
+    let uuid = Uuidm.v4_gen rand () in
     let dir_name = "semgrep_git_worktree_" ^ Uuidm.to_string uuid in
     let dir = CapTmp.get_temp_dir_name caps#tmp / dir_name in
     UUnix.mkdir !!dir 0o777;
diff --git a/libs/tracing/unix/Tracing.ml b/libs/tracing/unix/Tracing.ml
index 27d9073fc39a..bb3873b2db64 100644
--- a/libs/tracing/unix/Tracing.ml
+++ b/libs/tracing/unix/Tracing.ml
@@ -201,6 +201,18 @@ let trace_data_only ?(level = Info) ~__FUNCTION__ ~__FILE__ ~__LINE__ name
   with_span ~level ~__FUNCTION__ ~__FILE__ ~__LINE__ name (fun sp ->
       f () |> add_yojson_to_span sp)
 
+let log_trace_message () =
+  match Otel.Scope.get_ambient_scope () with
+  | None ->
+      (* nosemgrep: no-logs-in-library *)
+      Logs.info (fun m ->
+          m "Tracing is enabled for this scan. There was no trace id recorded.")
+  | Some scope ->
+      let id = Otel.Trace_id.to_hex scope.trace_id in
+      (* nosemgrep: no-logs-in-library *)
+      Logs.info (fun m ->
+          m "Tracing is enabled for this scan. The trace id is <%s>." id)
+
 (*****************************************************************************)
 (* Entry points for setting up tracing *)
 (*****************************************************************************)
@@ -241,7 +253,9 @@ let with_tracing fname trace_endpoint data f =
   Opentelemetry_client_ocurl.with_setup ~config () @@ fun () ->
   with_top_level_span ?parent_span_id ?parent_trace_id ~__FILE__ ~__LINE__ ~data
     fname
-  @@ fun sp -> f sp
+  @@ fun sp ->
+  log_trace_message ();
+  f sp
 
 (* Alt: using cohttp_lwt (we probably want to do this when we switch to Eio w/ *)
 (* their compatibility layer)
diff --git a/PRIVACY.md b/metrics.md
similarity index 96%
rename from PRIVACY.md
rename to metrics.md
index ac86eb05291f..c8ec031669d8 100644
--- a/PRIVACY.md
+++ b/metrics.md
@@ -1,6 +1,6 @@
-# Semgrep Privacy Policy
+# Semgrep metrics
 
-Semgrep may collect aggregate metrics to help improve the product. This document describes:
+Semgrep CLI may collect aggregate metrics to help improve the product. This document describes:
 
 - [the principles that guide our data-collection decisions](#principles)
 - [how to change when Semgrep sends metrics](#automatic-collection-opt-in-and-opt-out)
@@ -61,21 +61,21 @@ The following data will never leave your environment as part of metrics.
 - Git commit hashes, timestamps, messages, authors
 - User-identifiable data about Semgrep’s findings in your code, including finding messages
 
-This data will be sent to Semgrep App only if you explicitly request it,
-such as with `semgrep login && semgrep ci` to connect with Semgrep App.
+This data will be sent to Semgrep AppSec Platform only if you explicitly request it,
+such as with `semgrep login && semgrep ci` to connect with Semgrep AppSec Platform.
 Even in that case, your source code and private rules will never be sent.
 
 ## Data collected as metrics
 
-Semgrep collects data to improve the user experience.
+Semgrep CLI collects data to improve the user experience.
 Five types of data are collected:
 
 ### Environmental
 
-Environmental data provide contextual data about Semgrep’s runtime environment, as well as information that helps debug any issues users may be facing; e.g.
+Environmental data provide contextual data about Semgrep CLI’s runtime environment, as well as information that helps debug any issues users may be facing; e.g.
 
 - How long the command took to run
-- The version of Semgrep
+- The version of Semgrep CLI
 - An [anonymous user ID](#anonymous-user-id) that identifies the machine
 - IP address that triggers a run
 - Value of the CI environment variable, if set
@@ -86,7 +86,7 @@ Environmental data provide contextual data about Semgrep’s runtime environment
 
 ### Performance
 
-Performance data enable understanding of which rules and types of files are slow in the aggregate so Semgrep, Inc can improve the Semgrep program-analysis engine, query optimizer, and debug slow rules; e.g.
+Performance data enable understanding of which rules and types of files are slow in the aggregate so Semgrep, Inc can improve the program-analysis engine, query optimizer, and debug slow rules; e.g.
 
 - Runtime duration
 - Duration of individual phases (e.g. parsing)
@@ -96,7 +96,7 @@ Performance data enable understanding of which rules and types of files are slow
 
 ### Parse Rates
 
-Semgrep reports aggregated parse rate information on a per-language basis; e.g.,
+Aggregated parse rate information is reported on a per-language basis; e.g.,
 
 - Number of targeted files
 - Number of files without any parse-related error
@@ -105,7 +105,7 @@ Semgrep reports aggregated parse rate information on a per-language basis; e.g.,
 
 ### Errors
 
-High-level error and warning classes that Semgrep encounters when run; e.g.
+High-level error and warning classes encountered when run; e.g.
 
 - Semgrep’s return code
 - The number of errors
@@ -113,17 +113,17 @@ High-level error and warning classes that Semgrep encounters when run; e.g.
 
 ### Value
 
-Semgrep reports data that indicate how useful a run is for the end user; e.g.
+Data that indicate how useful a run is for the end user; e.g.
 
 - Number of raised findings
 - Number of ignored findings
 - Pseudoanonymized hashes of the rule definitions that yield findings
-- The [Semgrep features used](#feature-usage) during the scan
+- The [features used](#feature-usage) during the scan
 - The engine type requested for the scan
 
 ### Extension
 
-Semgrep reports additional data when used in conjunction with an IDE integration, such as the [Semgrep VSCode Extension](https://github.com/semgrep/semgrep-vscode), that help us understand what IDEs are used and how helpful the integrations are for users; e.g.
+Additional data is reported when used in conjunction with an IDE integration, such as the [Semgrep VS Code Extension](https://github.com/semgrep/semgrep-vscode), that help us understand what IDEs are used and how helpful the integrations are for users; e.g.
 
 - IDE being used
 - Version of IDE integration
@@ -343,14 +343,14 @@ This is a sample blob of the aggregate metrics described above:
 
 ## Data collected when explicitly requested
 
-For Semgrep App users running `semgrep ci` while logged in,
+For Semgrep AppSec Platform users running `semgrep ci` while logged in,
 data is sent to power your dashboard, notification, dependency search, and finding management features.
-These data are ONLY sent when using `semgrep ci` in an App-connected mode
+These data are ONLY sent when using `semgrep ci` in a platform-connected mode
 and are not sent when not logged in.
 
 Three types of data are sent to Semgrep, Inc servers for this logged-in use case: scan data, findings data, and dependencies data.
 
-**Scan data** provide information on the environment and performance of Semgrep.
+**Scan data** provide information on the environment and performance.
 They power dashboards, identify anomalies with the product, and are needed for billing.
 The classes of scan data are:
 
@@ -362,7 +362,7 @@ The classes of scan data are:
 - Scan metadata, including type of scan and scan parameters (e.g., paths scanned and extensions of ignored files)
 - Timing metrics (e.g., time taken to scan per-rule and per-path)
 - Parse metrics (e.g., number of files targeted and parsed per-language)
-- Semgrep environment (e.g., version, interpreter, timestamp)
+- Semgrep CLI environment (e.g., version, interpreter, timestamp)
 
 **Findings data** are used to provide human readable content for notifications and integrations,
 as well tracking results as new, fixed, or duplicate. The classes of findings data are:
@@ -382,12 +382,12 @@ dependencies data are:
 
 ## Debugging data collected when traces are requested
 
-To help debug performance issues, Semgrep can send traces, enabled via `--trace`.
+To help debug performance issues, Semgrep CLI can send traces, enabled via `--trace`.
 Traces are never sent unless the `--trace` flag is included.
 
 There are three modes of tracing.
 1. Info (`--trace`): basic tracing. Sends timings about each file as it undergoes pre-processing and then matching. Includes the file path and sometimes rule names.
-2. Debug (`--trace` with `SEMGREP_TRACE_LEVEL=debug`): debug tracing. Sends additional timings, particularly around functions Semgrep runs during taint analysis.
+2. Debug (`--trace` with `SEMGREP_TRACE_LEVEL=debug`): debug tracing. Sends additional timings, particularly around functions run during taint analysis.
 3. Trace (`--trace` with `SEMGREP_TRACE_LEVEL=trace`): even more detailed debug tracing.
 
 All traces are sent in Opentelemetry format and may include:
diff --git a/release_changes.md b/release_changes.md
index efaafe61d76d..255052d565fd 100644
--- a/release_changes.md
+++ b/release_changes.md
@@ -1,27 +1,24 @@
-## [1.90.0](https://github.com/returntocorp/semgrep/releases/tag/v1.90.0) - 2024-09-25
+## [1.91.0](https://github.com/returntocorp/semgrep/releases/tag/v1.91.0) - 2024-10-10
 
 
 ### Added
 
 
-- Expanded support for requirement lockfiles. Semgrep will now find any `*requirement*.txt`
-  file and lockfiles in a requirements folder (`**/requirements/*.txt`). This functionality
-  will be gated behind the `--enable-experimental-requirements` CLI flag. (sc-1752)
-
-
-### Changed
-
-
-- Security update for code snippet storage & access methods. (gh-2038)
+- Type inference in the Pro engine has been improved for class fields in
+  TypeScript that are assigned a new instance but lack an explicit type
+  definition. When no explicit type is provided for a class field, its type is
+  inferred from the type of the expression assigned to it. For example, in the
+  class definition `class Foo { private readonly bar = new Bar(); }`, the type of
+  `bar` is inferred to be `Bar`. (code-7635)
+- Cargo.lock parser can now associate dependencies with lockfile line numbers (sc-1140)
 
 
 ### Fixed
 
 
-- Errors that occur in semgrep scans with jobs > 1 will now have more detail (SAF-1628)
-- Dockerfile matching: `CMD $...ARGS` now behaves like `CMD ...` and matches
-  any CMD instruction that uses the array syntax such as `CMD ["ls"]`. This
-  fix also applies to the other command-like instructions RUN
-  and ENTRYPOINT. (gh-9726)
-- Pro Engine: There is now improved type inference in Kotlin and Scala. Constructor invocations like
-  `Foo()` will now be inferred properly to be of type `Foo`. (saf-1537)
+- Address python `rich.errors.LiveError` where attempting to display multiple progress bars
+  raises an exception as flagged in #10562. (grow-414)
+- C: Fix a regression causing pattern `-n` to sometimes not match code `-n`. (saf-1592)
+- When a scan runs into an exception, the app is appropriately notified
+  about the failure. Previously, in the app, it would seem to the user
+  that the scan is still in progress. (sms-502)
diff --git a/semgrep.opam b/semgrep.opam
index c0497637d6d8..9d107e9a8a04 100644
--- a/semgrep.opam
+++ b/semgrep.opam
@@ -1,6 +1,6 @@
 # This file is generated by dune, edit dune-project instead
 opam-version: "2.0"
-version: "1.79.0"
+version: "1.90.0"
 synopsis:
   "Like grep but for code: fast and syntax-aware semantic code pattern for many languages"
 description: """
@@ -12,7 +12,7 @@ For more information see https://semgrep.dev
 """
 maintainer: ["Yoann Padioleau <pad@semgrep.com>"]
 authors: ["Yoann Padioleau <pad@semgrep.com>"]
-license: "LGPL-2.1"
+license: "LGPL-2.1-only"
 homepage: "https://semgrep.dev"
 bug-reports: "https://github.com/semgrep/semgrep/issues"
 depends: [
@@ -65,7 +65,7 @@ depends: [
   "ambient-context-lwt"
   "conf-libcurl" {= "1"}
   "uri"
-  "uuidm"
+  "uuidm" {>= "0.9.9"}
   "cohttp" {= "5.3.0"}
   "cohttp-lwt-unix"
   "cohttp-lwt-jsoo"
diff --git a/setup.py b/setup.py
index e25356008210..94dc831d45ff 100644
--- a/setup.py
+++ b/setup.py
@@ -5,7 +5,7 @@
 
 setup(
     name="semgrep_pre_commit_package",
-    version="1.90.0",
-    install_requires=["semgrep==1.90.0"],
+    version="1.91.0",
+    install_requires=["semgrep==1.91.0"],
     packages=[],
 )
diff --git a/src/analyzing/AST_to_IL.ml b/src/analyzing/AST_to_IL.ml
index 10347dab74fc..24d444af69e0 100644
--- a/src/analyzing/AST_to_IL.ml
+++ b/src/analyzing/AST_to_IL.ml
@@ -1745,6 +1745,7 @@ and stmt_aux env st =
   | G.OtherStmt _
   | G.OtherStmtWithStmt _ ->
       todo (G.S st)
+  | G.RawStmt _ -> todo (G.S st)
 
 and for_each env tok (pat, tok2, e) st =
   let cont_label_s, break_label_s, st_env = mk_break_continue_labels env tok in
diff --git a/src/core/Version.ml b/src/core/Version.ml
index 7a30f4b368c3..267f67092f3c 100644
--- a/src/core/Version.ml
+++ b/src/core/Version.ml
@@ -3,4 +3,4 @@
 
   Automatically modified by scripts/release/bump.
 *)
-let version = "1.90.0"
+let version = "1.91.0"
diff --git a/src/engine/Formula_cache.ml b/src/engine/Formula_cache.ml
new file mode 100644
index 000000000000..47224d2e8b64
--- /dev/null
+++ b/src/engine/Formula_cache.ml
@@ -0,0 +1,96 @@
+(* Brandon Wu
+ *
+ * Copyright (C) 2023-present Semgrep Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public License
+ * version 2.1 as published by the Free Software Foundation, with the
+ * special exception on linking described in file LICENSE.
+ *
+ * This library is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the file
+ * LICENSE for more details.
+ *)
+module Log = Log_analyzing.Log
+module R = Rule
+module RM = Range_with_metavars
+module ME = Matching_explanation
+
+(* This Formula_tbl structure is used to create a "formula cache", which will
+   permit the sharing of matches resulting from common formulas that occur as
+   sources, sinks, sanitizers, or propagators.
+
+   In particular, because of hardcoded propagators, we expect to see lots of
+   sharing from Semgrep Pro Engine.
+*)
+module Formula_tbl = Hashtbl.Make (struct
+  type t = R.formula
+
+  let equal = R.equal_formula
+  let hash = R.hash_formula
+end)
+
+type formula_matches = RM.t list * ME.t list
+type t = (formula_matches option * int) Formula_tbl.t
+
+(* This function is for creating a formula cache which only caches formula that
+   it knows will be shared, at least once, among the formula in a bunch of
+   taint rules.
+
+   This is because it's obviously not useful to cache a formula's matches if
+   that formula never comes up again. This cache stores an option, with keys
+   that are only formula that are guaranteed to appear more than once in the
+   collection.
+*)
+let mk_specialized_formula_cache (rules : R.taint_rule list) =
+  let count_tbl = Formula_tbl.create 128 in
+  let flat_formulas =
+    rules
+    |> List.concat_map (fun (rule : R.taint_mode R.rule_info) ->
+           let (`Taint (spec : R.taint_spec)) = rule.R.mode in
+           R.formula_of_mode (`Taint spec))
+  in
+  flat_formulas
+  |> List.iter (fun formula ->
+         match Formula_tbl.find_opt count_tbl formula with
+         | None -> Formula_tbl.add count_tbl formula (None, 1)
+         | Some (_, x) -> Formula_tbl.replace count_tbl formula (None, 1 + x));
+  (* We return the table with pairs of (None, count) itself.
+     When we try to cache a find, we will first check whether decreasing this
+     counter results in 0. Then, there are no more uses, and the result is no
+     longer worth caching.
+     This way we don't keep around entries when we don't need to.
+  *)
+  count_tbl
+
+let cached_find_opt formula_cache formula ~get_matches =
+  match Formula_tbl.find_opt formula_cache formula with
+  | None ->
+      (* it should not actually be possible for a formula to
+         not be in the formula table
+
+         just don't cache it I guess
+      *)
+      Log.warn (fun m ->
+          m
+            "Tried to compute matches for a formula not in the cache \
+             (impossible?)");
+      get_matches ()
+  | Some (None, count) ->
+      let ranges, expls = get_matches () in
+      if count <= 1 then
+        (* if there's only 1 more use left, there's no point
+           in caching it
+        *)
+        (ranges, expls)
+      else (
+        (* otherwise, this is the first time we've seen this
+           formula, and we should cache it
+        *)
+        Formula_tbl.replace formula_cache formula
+          (Some (ranges, expls), count - 1);
+        (ranges, expls))
+  | Some (Some (ranges, expls), count) ->
+      if count <= 1 then Formula_tbl.remove formula_cache formula;
+      (ranges, expls)
diff --git a/src/engine/Formula_cache.mli b/src/engine/Formula_cache.mli
new file mode 100644
index 000000000000..29ed4de8f5ab
--- /dev/null
+++ b/src/engine/Formula_cache.mli
@@ -0,0 +1,18 @@
+type formula_matches = Range_with_metavars.t list * Matching_explanation.t list
+
+type t
+(**
+   The type of the specialized formual cache used for inter-rule
+   match sharing.
+*)
+
+val mk_specialized_formula_cache : Rule.taint_rule list -> t
+(**
+   These formula caches are only safe to use to share results between
+   runs of rules on the same target!
+
+   Right now they are only used by [Match_tainting_mode.taint_config_of_rule].
+*)
+
+val cached_find_opt :
+  t -> Rule.formula -> get_matches:(unit -> formula_matches) -> formula_matches
diff --git a/src/engine/Match_taint_spec.ml b/src/engine/Match_taint_spec.ml
new file mode 100644
index 000000000000..fe1f5ed561bb
--- /dev/null
+++ b/src/engine/Match_taint_spec.ml
@@ -0,0 +1,501 @@
+(* Iago Abal, Yoann Padioleau
+ *
+ * Copyright (C) 2019-2024 Semgrep Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public License
+ * version 2.1 as published by the Free Software Foundation, with the
+ * special exception on linking described in file LICENSE.
+ *
+ * This library is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the file
+ * LICENSE for more details.
+ *)
+open Common
+open Fpath_.Operators (* e.g. !! *)
+module Log = Log_tainting.Log
+module G = AST_generic
+module ME = Matching_explanation
+module MV = Metavariable
+module R = Rule
+module RM = Range_with_metavars
+module OutJ = Semgrep_output_v1_t
+module D = Dataflow_tainting
+
+(* Taint-tracking via ranges
+ * -------------------------
+ *
+ * First we run a bunch of search queries to obtain the ranges of sources,
+ * sanitizers, and sinks. The actual analysis happens in Dataflow_tainting
+ * using the IL representation. There we check whether the eorig's and iorig's
+ * are within those ranges to decide whether an expression or instruction is a
+ * source of taint, a sanitizer, or a sink. Finally we collect the results
+ * produced by the dataflow analysis and filter out duplicates.
+ *
+ * We could think of using an equality test to compare ranges, e.g., an
+ * expression would be a source of taint only if its range is exactly one of
+ * the ranges matched by `pattern-sources`. In practice, this does not work
+ * because `pattern-sources` etc can match anything, and the IL eorig's and
+ * iorig's are only expressions. For example, `pattern-sources` can match
+ * `foo(x);` but the eorig in the IL will be `foo(x)` whose range does not
+ * include the ending `;`.
+ *
+ * So, we use sub-range checks. And this actually provides some extra power,
+ * as it allows us to mark anything as a source/sanitizer/sink. For example,
+ * we could use a pattern like `if (E) { ... }` to specify that anything
+ * inside such an `if` statement should be considered sanitized. We are not
+ * limited to expressions or single statements.
+ *
+ * However, using sub-range checks leads to duplicates. For example, the PHP
+ * expression `sink("$source" . 'here')` will be translated to IL as two
+ * instructions `tmp = "$source" . 'here'` and `sink(tmp)`. If `sink(...)`
+ * is a `pattern-sinks`, then both instructions' ranges are inside
+ * the `pattrn-sinks` ranges. If `$source` is a `pattern-sources`, then both
+ * instructions are also tainted, and Dataflow_tainting will report two
+ * matches.
+ *
+ * So, we need to remove duplicate subamtches at the end.
+ * TODO: We could perhaps do this in a cleaner way by having an intermediate
+ * step where we annotate the Generic AST, marking which statements and
+ * expressions are sources, sanitizers, or sinks. If e.g. an expression is a
+ * sink, we take care not to mark as sinks any of its subexpressions, in order
+ * to prevent duplicates.
+ *)
+
+type propagator_match = {
+  id : D.var;
+  rwm : RM.t;
+  from : Range.t;
+  to_ : Range.t;
+  spec : R.taint_propagator;
+}
+
+type spec_matches = {
+  sources : (RM.t * R.taint_source) list;
+  propagators : propagator_match list;
+  sanitizers : (RM.t * R.taint_sanitizer) list;
+  sinks : (RM.t * R.taint_sink) list;
+}
+
+(*****************************************************************************)
+(* Finding matches for taint specs *)
+(*****************************************************************************)
+
+(* Finds all matches of a taint-spec pattern formula. *)
+let range_w_metas_of_formula (xconf : Match_env.xconfig) (xtarget : Xtarget.t)
+    (rule : R.t) (formula : R.formula) : RM.ranges * ME.t list =
+  (* !! Calling Match_search_mode here !! *)
+  let report, ranges =
+    Match_search_mode.matches_of_formula xconf rule xtarget formula None
+  in
+  (ranges, report.explanations)
+
+(* =~ List.concat_map with automatic management of matching-explanations *)
+let concat_map_with_expls f xs =
+  let all_expls = ref [] in
+  let res =
+    xs
+    |> List.concat_map (fun x ->
+           let ys, expls = f x in
+           Stack_.push expls all_expls;
+           ys)
+  in
+  (res, List_.flatten (List.rev !all_expls))
+
+let%test _ =
+  concat_map_with_expls (fun x -> ([ -x; x ], [ 2 * x; 3 * x ])) [ 0; 1; 2 ]
+  =*= ([ 0; 0; -1; 1; -2; 2 ], [ 0; 0; 2; 3; 4; 6 ])
+
+let find_range_w_metas formula_cache (xconf : Match_env.xconfig)
+    (xtarget : Xtarget.t) (rule : R.t) (specs : (R.formula * 'a) list) :
+    (RM.t * 'a) list * ME.t list =
+  (* TODO: Make an Or formula and run a single query. *)
+  (* if perf is a problem, we could build an interval set here *)
+  specs
+  |> concat_map_with_expls (fun (pf, x) ->
+         let ranges, expls =
+           Formula_cache.cached_find_opt formula_cache pf (fun () ->
+               range_w_metas_of_formula xconf xtarget rule pf)
+         in
+         (ranges |> List_.map (fun rwm -> (rwm, x)), expls))
+
+let find_sanitizers_matches formula_cache (xconf : Match_env.xconfig)
+    (xtarget : Xtarget.t) (rule : R.t) (specs : R.taint_sanitizer list) :
+    (bool * RM.t * R.taint_sanitizer) list * ME.t list =
+  specs
+  |> concat_map_with_expls (fun (sanitizer : R.taint_sanitizer) ->
+         let ranges, expls =
+           Formula_cache.cached_find_opt formula_cache
+             sanitizer.sanitizer_formula (fun () ->
+               range_w_metas_of_formula xconf xtarget rule
+                 sanitizer.sanitizer_formula)
+         in
+         ( ranges
+           |> List_.map (fun x -> (sanitizer.R.not_conflicting, x, sanitizer)),
+           expls ))
+
+(* Finds all matches of `pattern-propagators`. *)
+let find_propagators_matches formula_cache (xconf : Match_env.xconfig)
+    (xtarget : Xtarget.t) (rule : R.t)
+    (propagators_spec : R.taint_propagator list) =
+  propagators_spec
+  |> List.concat_map (fun (p : R.taint_propagator) ->
+         let mvar_pfrom, tok_pfrom = p.from in
+         let mvar_pto, tok_pto = p.to_ in
+         let ranges_w_metavars, _expsTODO =
+           Formula_cache.cached_find_opt formula_cache p.propagator_formula
+             (fun () ->
+               range_w_metas_of_formula xconf xtarget rule p.propagator_formula)
+         in
+         (* Now, for each match of the propagator pattern, we try to construct
+          * a `propagator_match`. We just need to look up what code is captured
+          * by the metavariables `from` and `to`, and check if we can obtain good
+          * location info for that code (i.e., we have real tokens rather than
+          * fake ones). *)
+         ranges_w_metavars
+         |> List_.filter_map (fun rwm ->
+                (* The piece of code captured by the `from` metavariable.  *)
+                let* _mvar_from, mval_from =
+                  List.find_opt
+                    (fun (mvar, _mval) -> MV.equal_mvar mvar_pfrom mvar)
+                    rwm.RM.mvars
+                in
+                (* The piece of code captured by the `to` metavariable.  *)
+                let* _mvar_to, mval_to =
+                  List.find_opt
+                    (fun (mvar, _mval) -> MV.equal_mvar mvar_pto mvar)
+                    rwm.RM.mvars
+                in
+                (* TODO: log a warning when we cannot obtain a taint propagator due to
+                 * lacking range info. *)
+                match (Tok.loc_of_tok tok_pfrom, Tok.loc_of_tok tok_pto) with
+                | Error _, _
+                | _, Error _ ->
+                    None
+                | Ok loc_pfrom, Ok loc_pto ->
+                    let* mval_from_start_loc, mval_from_end_loc =
+                      AST_generic_helpers.range_of_any_opt
+                        (MV.mvalue_to_any mval_from)
+                    in
+                    let* mval_to_start_loc, mval_to_end_loc =
+                      AST_generic_helpers.range_of_any_opt
+                        (MV.mvalue_to_any mval_to)
+                    in
+                    let from =
+                      Range.range_of_token_locations mval_from_start_loc
+                        mval_from_end_loc
+                    in
+                    let to_ =
+                      Range.range_of_token_locations mval_to_start_loc
+                        mval_to_end_loc
+                    in
+                    let id =
+                      Common.spf "propagator:%d:%d:%d:%d:%d:%d"
+                        loc_pfrom.pos.bytepos loc_pto.pos.bytepos
+                        from.Range.start from.Range.end_ to_.Range.start
+                        to_.Range.end_
+                    in
+                    Some { id; rwm; from; to_; spec = p }))
+
+(*****************************************************************************)
+(* Spec matches *)
+(*****************************************************************************)
+
+let spec_matches_of_taint_rule ~per_file_formula_cache xconf file ast_and_errors
+    ({ mode = `Taint spec; _ } as rule : R.taint_rule) =
+  let file = Fpath.v file in
+  let formula_cache = per_file_formula_cache in
+  let xconf = Match_env.adjust_xconfig_with_rule_options xconf rule.options in
+  let lazy_ast_and_errors = lazy ast_and_errors in
+  (* TODO: should this function just take a target, rather than a file? *)
+  let xtarget : Xtarget.t =
+    {
+      path = { origin = File file; internal_path_to_content = file };
+      xlang = rule.target_analyzer;
+      lazy_content = lazy (UFile.read_file file);
+      lazy_ast_and_errors;
+    }
+  in
+  let (sources_ranges : (RM.t * R.taint_source) list), expls_sources =
+    find_range_w_metas formula_cache xconf xtarget rule
+      (spec.sources |> snd
+      |> List_.map (fun (src : R.taint_source) -> (src.source_formula, src)))
+  and (propagators_ranges : propagator_match list) =
+    find_propagators_matches formula_cache xconf xtarget rule spec.propagators
+  and (sinks_ranges : (RM.t * R.taint_sink) list), expls_sinks =
+    find_range_w_metas formula_cache xconf xtarget rule
+      (spec.sinks |> snd
+      |> List_.map (fun (sink : R.taint_sink) -> (sink.sink_formula, sink)))
+  in
+  let sanitizers_ranges, expls_sanitizers =
+    match spec.sanitizers with
+    | None -> ([], [])
+    | Some (_, sanitizers_spec) ->
+        find_sanitizers_matches formula_cache xconf xtarget rule sanitizers_spec
+  in
+  let (sanitizers_ranges : (RM.t * R.taint_sanitizer) list) =
+    (* A sanitizer cannot conflict with a sink or a source, otherwise it is
+     * filtered out. This allows to e.g. declare `$F(...)` as a sanitizer,
+     * to assume that any other function will handle tainted data safely.
+     * Without this, `$F(...)` will automatically sanitize any other function
+     * call acting as a sink or a source. *)
+    sanitizers_ranges
+    |> List_.filter_map (fun (not_conflicting, range, spec) ->
+           (* TODO: Warn user when we filter out a sanitizer? *)
+           if not_conflicting then
+             if
+               not
+                 (List.exists
+                    (fun (range', _) -> range'.RM.r =*= range.RM.r)
+                    sinks_ranges
+                 || List.exists
+                      (fun (range', _) -> range'.RM.r =*= range.RM.r)
+                      sources_ranges)
+             then Some (range, spec)
+             else None
+           else Some (range, spec))
+  in
+  let expls =
+    if xconf.matching_explanations then
+      let ranges_to_pms ranges_and_stuff =
+        ranges_and_stuff
+        |> List_.map (fun (rwm, _) ->
+               RM.range_to_pattern_match_adjusted rule rwm)
+      in
+      [
+        {
+          ME.op = OutJ.TaintSource;
+          pos = fst spec.sources;
+          children = expls_sources;
+          matches = ranges_to_pms sources_ranges;
+          extra = None;
+        };
+        {
+          ME.op = OutJ.TaintSink;
+          pos = fst spec.sinks;
+          children = expls_sinks;
+          matches = ranges_to_pms sinks_ranges;
+          extra = None;
+        }
+        (* TODO: propagators *);
+      ]
+      @
+      match spec.sanitizers with
+      | None -> []
+      | Some (tok, _) ->
+          [
+            {
+              ME.op = OutJ.TaintSanitizer;
+              pos = tok;
+              children = expls_sanitizers;
+              (* 'sanitizer_ranges' will be affected by `not-conflicting: true`:
+               * if a sanitizer coincides exactly with a source/sink then it will
+               * be filtered out. So the sanitizer matches may not be the union of
+               * the matches of the individual sanitizers. Anyhow, not-conflicting
+               * has been deprecated for quite some time, and we will remove it at
+               * some point. *)
+              matches = ranges_to_pms sanitizers_ranges;
+              extra = None;
+            };
+          ]
+    else []
+  in
+  ( {
+      sources = sources_ranges;
+      propagators = propagators_ranges;
+      sanitizers = sanitizers_ranges;
+      sinks = sinks_ranges;
+    },
+    expls )
+
+(*****************************************************************************)
+(* Testing whether a an AST node matches a taint spec *)
+(*****************************************************************************)
+
+let option_bind_list opt f =
+  match opt with
+  | None -> []
+  | Some x -> f x
+
+module Range_table = struct
+  module T = Hashtbl.Make (struct
+    type t = Range.t
+
+    let equal = Range.equal
+    let hash = Hashtbl.hash
+  end)
+
+  let create () = T.create 100
+
+  let push tbl k v =
+    match T.find_opt tbl k with
+    | None -> T.add tbl k [ v ]
+    | Some vs -> T.replace tbl k (v :: vs)
+
+  let get tbl k =
+    match T.find_opt tbl k with
+    | None -> []
+    | Some vs -> vs
+end
+
+let range_of_any any =
+  (* This is potentially slow. We may need to store range position in
+   * the AST at some point. *)
+  match AST_generic_helpers.range_of_any_opt any with
+  | None ->
+      (* IL.any_of_orig will return `G.Anys []` for `NoOrig`, and there is
+       * no point in issuing this warning in that case.
+       * TODO: Perhaps we should avoid the call to `any_in_ranges` in the
+       * first place? *)
+      if any <> G.Anys [] then
+        Log.debug (fun m ->
+            m "Cannot compute range, there are no real tokens in this AST: %s"
+              (G.show_any any));
+      None
+  | Some (tok1, tok2) ->
+      let r = Range.range_of_token_locations tok1 tok2 in
+      Some r
+
+(* Assuming that `r` is a subrange of `match_range` then this computes a
+ * float in [0.0, 1.0]. We expect `r` to be the range of some arbitrary
+ * piece of code,  and `match_range` to be the range of a match of some
+ * taint spec (e.g. a taint source). Then this float indicates how much
+ * overlap there is between the code and the spec. The degree of overlap
+ * is used to determine whether the match is "exact" (overlap > 0.99),
+ * which e.g. triggets the side-effectful propagation of taint. *)
+let overlap_with ~match_range r =
+  let r1 = match_range in
+  float_of_int (r.Range.end_ - r.Range.start + 1)
+  /. float_of_int (r1.Range.end_ - r1.Range.start + 1)
+
+let any_is_in_sources_matches rule any matches =
+  let ( let* ) = option_bind_list in
+  let* r = range_of_any any in
+  matches
+  |> List_.filter_map (fun (rwm, (ts : R.taint_source)) ->
+         if Range.( $<=$ ) r rwm.RM.r then
+           Some
+             (let spec_pm = RM.range_to_pattern_match_adjusted rule rwm in
+              let overlap = overlap_with ~match_range:rwm.RM.r r in
+              {
+                Taint_spec_match.spec = ts;
+                spec_id = ts.source_id;
+                spec_pm;
+                range = r;
+                overlap;
+              })
+         else None)
+
+(* Builds a table to quickly look up for propagator matches, taking advantage of
+ * the requirement that propagators must be exact matches.
+ *
+ * OBS: Previously we allowed these matches if they had an overlap of >0.99, but
+ *   to be honest that was arbitrary and fragile (the overlap largely depends on
+ *   the amount of text being matched). Typically propagators from/to match
+ *   l-values and those typically we can count as being 100% perfect matches.
+ *   If this causes problems we can always roll back, but all tests still work.
+ *)
+let propagators_table_of_matches rule matches =
+  let mk_match (prop : propagator_match) var kind r =
+    let spec_pm = RM.range_to_pattern_match_adjusted rule prop.rwm in
+    let spec : D.a_propagator = { kind; prop = prop.spec; var } in
+    {
+      Taint_spec_match.spec;
+      spec_id = prop.spec.propagator_id;
+      spec_pm;
+      range = r;
+      overlap = 1.0;
+    }
+  in
+  let tbl = Range_table.create () in
+  matches
+  |> List.iter (fun (prop : propagator_match) ->
+         let var = prop.id in
+         Range_table.push tbl prop.to_ (mk_match prop var `To prop.to_);
+         Range_table.push tbl prop.from (mk_match prop var `From prop.from));
+  tbl
+
+(* Check whether `any` matches either the `from` or the `to` of any of the
+ * `pattern-propagators`. Matches must be exact to make taint propagation
+ * more precise and predictable.
+ *
+ * THINK: Now that we have "Best_matches" we could perhaps use that for
+ *   propagators too?
+ *)
+let any_is_in_propagators_matches any tbl :
+    D.a_propagator Taint_spec_match.t list =
+  match range_of_any any with
+  | None -> []
+  | Some r -> Range_table.get tbl r
+
+let any_is_in_sanitizers_matches rule any matches =
+  let ( let* ) = option_bind_list in
+  let* r = range_of_any any in
+  matches
+  |> List_.filter_map (fun (rwm, spec) ->
+         if Range.( $<=$ ) r rwm.RM.r then
+           Some
+             (let spec_pm = RM.range_to_pattern_match_adjusted rule rwm in
+              let overlap = overlap_with ~match_range:rwm.RM.r r in
+              {
+                Taint_spec_match.spec;
+                spec_id = spec.R.sanitizer_id;
+                spec_pm;
+                range = r;
+                overlap;
+              })
+         else None)
+
+let any_is_in_sinks_matches rule any matches =
+  let ( let* ) = option_bind_list in
+  let* r = range_of_any any in
+  matches
+  |> List_.filter_map (fun (rwm, (spec : R.taint_sink)) ->
+         if Range.( $<=$ ) r rwm.RM.r then
+           Some
+             (let spec_pm = RM.range_to_pattern_match_adjusted rule rwm in
+              let overlap = overlap_with ~match_range:rwm.RM.r r in
+              {
+                Taint_spec_match.spec;
+                spec_id = spec.sink_id;
+                spec_pm;
+                range = r;
+                overlap;
+              })
+         else None)
+
+(*****************************************************************************)
+(* Entry point *)
+(*****************************************************************************)
+
+let taint_config_of_rule ~per_file_formula_cache xconf file ast_and_errors
+    ({ mode = `Taint spec; _ } as rule : R.taint_rule) handle_effects =
+  let spec_matches, expls =
+    spec_matches_of_taint_rule ~per_file_formula_cache xconf file ast_and_errors
+      rule
+  in
+  let file = Fpath.v file in
+  let xconf = Match_env.adjust_xconfig_with_rule_options xconf rule.options in
+  let propagators_tbl =
+    propagators_table_of_matches rule spec_matches.propagators
+  in
+  let config = xconf.config in
+  ( {
+      Dataflow_tainting.filepath = !!file;
+      rule_id = fst rule.R.id;
+      track_control =
+        spec.sources |> snd
+        |> List.exists (fun (src : R.taint_source) -> src.source_control);
+      is_source =
+        (fun x -> any_is_in_sources_matches rule x spec_matches.sources);
+      is_propagator = (fun x -> any_is_in_propagators_matches x propagators_tbl);
+      is_sanitizer =
+        (fun x -> any_is_in_sanitizers_matches rule x spec_matches.sanitizers);
+      is_sink = (fun x -> any_is_in_sinks_matches rule x spec_matches.sinks);
+      unify_mvars = config.taint_unify_mvars;
+      handle_effects;
+    },
+    spec_matches,
+    expls )
+[@@trace_trace]
diff --git a/src/engine/Match_taint_spec.mli b/src/engine/Match_taint_spec.mli
new file mode 100644
index 000000000000..06ba6d2b3114
--- /dev/null
+++ b/src/engine/Match_taint_spec.mli
@@ -0,0 +1,36 @@
+type propagator_match = {
+  id : Dataflow_tainting.var;
+      (** An unique identifier for the propagator match. This is used as an
+   * auxiliary variable to store the taints flowing from `from` to `to`. *)
+  rwm : Range_with_metavars.t;
+  from : Range.t;  (** The range matched by the `from` metavariable. *)
+  to_ : Range.t;  (** The range matched by the `to` metavariable. *)
+  spec : Rule.taint_propagator;
+}
+(** Taint will flow from `from` to `to_` through the axiliary variable `id`. *)
+
+type spec_matches = {
+  sources : (Range_with_metavars.t * Rule.taint_source) list;
+      (** Ranges matched by `pattern-sources:` *)
+  propagators : propagator_match list;
+      (** Ranges matched by `pattern-propagators:` *)
+  sanitizers : (Range_with_metavars.t * Rule.taint_sanitizer) list;
+      (** Ranges matched by `pattern-sanitizers:` *)
+  sinks : (Range_with_metavars.t * Rule.taint_sink) list;
+      (** Ranges matched by `pattern-sinks:` *)
+}
+
+(* It could be a private function, but it is also used by Deep Semgrep. *)
+(* This [formula_cache] argument is exposed here because this function is also
+   a subroutine but the cache itself should be created outside of the any main
+   loop which runs over rules. This cache is only safe to share with if
+   [taint_config_of_rule] is used on the same file!
+*)
+val taint_config_of_rule :
+  per_file_formula_cache:Formula_cache.t ->
+  Match_env.xconfig ->
+  string (* filename *) ->
+  AST_generic.program * Tok.location list ->
+  Rule.taint_rule ->
+  Dataflow_tainting.effects_handler ->
+  Dataflow_tainting.config * spec_matches * Matching_explanation.t list
diff --git a/src/engine/Match_tainting_mode.ml b/src/engine/Match_tainting_mode.ml
index cfa82ced77de..c111ff8a1182 100644
--- a/src/engine/Match_tainting_mode.ml
+++ b/src/engine/Match_tainting_mode.ml
@@ -20,7 +20,6 @@ module G = AST_generic
 module H = AST_generic_helpers
 module R = Rule
 module PM = Pattern_match
-module RM = Range_with_metavars
 module RP = Core_result
 module T = Taint
 module Lval_env = Taint_lval_env
@@ -34,77 +33,7 @@ module Effect = Shape_and_sig.Effect
 (*****************************************************************************)
 (* Prelude *)
 (*****************************************************************************)
-(* Wrapper around the tainting dataflow-based analysis.
- *
- * Here we pass matcher functions that uses semgrep patterns to
- * describe the source/sink/sanitizers.
- *
- * Taint-tracking via ranges
- * -------------------------
- *
- * First we run a bunch of search queries to obtain the ranges of sources,
- * sanitizers, and sinks. The actual analysis happens in Dataflow_tainting
- * using the IL representation. There we check whether the eorig's and iorig's
- * are within those ranges to decide whether an expression or instruction is a
- * source of taint, a sanitizer, or a sink. Finally we collect the results
- * produced by the dataflow analysis and filter out duplicates.
- *
- * We could think of using an equality test to compare ranges, e.g., an
- * expression would be a source of taint only if its range is exactly one of
- * the ranges matched by `pattern-sources`. In practice, this does not work
- * because `pattern-sources` etc can match anything, and the IL eorig's and
- * iorig's are only expressions. For example, `pattern-sources` can match
- * `foo(x);` but the eorig in the IL will be `foo(x)` whose range does not
- * include the ending `;`.
- *
- * So, we use sub-range checks. And this actually provides some extra power,
- * as it allows us to mark anything as a source/sanitizer/sink. For example,
- * we could use a pattern like `if (E) { ... }` to specify that anything
- * inside such an `if` statement should be considered sanitized. We are not
- * limited to expressions or single statements.
- *
- * However, using sub-range checks leads to duplicates. For example, the PHP
- * expression `sink("$source" . 'here')` will be translated to IL as two
- * instructions `tmp = "$source" . 'here'` and `sink(tmp)`. If `sink(...)`
- * is a `pattern-sinks`, then both instructions' ranges are inside
- * the `pattrn-sinks` ranges. If `$source` is a `pattern-sources`, then both
- * instructions are also tainted, and Dataflow_tainting will report two
- * matches.
- *
- * So, we need to remove duplicate subamtches at the end.
- * TODO: We could perhaps do this in a cleaner way by having an intermediate
- * step where we annotate the Generic AST, marking which statements and
- * expressions are sources, sanitizers, or sinks. If e.g. an expression is a
- * sink, we take care not to mark as sinks any of its subexpressions, in order
- * to prevent duplicates.
- *)
-
-type debug_taint = {
-  sources : (RM.t * R.taint_source) list;
-  sanitizers : RM.ranges;
-  sinks : (RM.t * R.taint_sink) list;
-}
-
-module Range_table = struct
-  module T = Hashtbl.Make (struct
-    type t = Range.t
-
-    let equal = Range.equal
-    let hash = Hashtbl.hash
-  end)
-
-  let create () = T.create 100
-
-  let push tbl k v =
-    match T.find_opt tbl k with
-    | None -> T.add tbl k [ v ]
-    | Some vs -> T.replace tbl k (v :: vs)
-
-  let get tbl k =
-    match T.find_opt tbl k with
-    | None -> []
-    | Some vs -> vs
-end
+(* Wrapper around the tainting dataflow-based analysis. *)
 
 (*****************************************************************************)
 (* Hooks *)
@@ -125,353 +54,14 @@ module DataflowY = Dataflow_core.Make (struct
   let short_string_of_node n = Display_IL.short_string_of_node_kind n.F2.n
 end)
 
-let option_bind_list opt f =
-  match opt with
-  | None -> []
-  | Some x -> f x
-
-(* Finds all matches of a taint-spec pattern formula. *)
-let range_w_metas_of_formula (xconf : Match_env.xconfig) (xtarget : Xtarget.t)
-    (rule : R.t) (formula : R.formula) : RM.ranges * ME.t list =
-  (* !! Calling Match_search_mode here !! *)
-  let report, ranges =
-    Match_search_mode.matches_of_formula xconf rule xtarget formula None
-  in
-  (ranges, report.explanations)
-
 let get_source_requires src =
   let _pm, src_spec = T.pm_of_trace src.T.call_trace in
   src_spec.R.source_requires
 
-type propagator_match = {
-  id : D.var;
-      (** An unique identifier for the propagator match. This is used as an
-   * auxiliary variable to store the taints flowing from `from` to `to`. *)
-  rwm : RM.t;
-  from : Range.t;  (** The range matched by the `from` metavariable. *)
-  to_ : Range.t;  (** The range matched by the `to` metavariable. *)
-  spec : R.taint_propagator;
-}
-(** Taint will flow from `from` to `to_` through the axiliary variable `id`. *)
-
-(* This Formula_tbl structure is used to create a "formula cache", which will
-   permit the sharing of matches resulting from common formulas that occur as
-   sources, sinks, sanitizers, or propagators.
-
-   In particular, because of hardcoded propagators, we expect to see lots of
-   sharing from Semgrep Pro Engine.
-*)
-module Formula_tbl = struct
-  include Hashtbl.Make (struct
-    type t = R.formula
-
-    let equal = R.equal_formula
-    let hash = R.hash_formula
-  end)
-
-  let cached_find_opt formula_cache formula compute_matches_fn =
-    match find_opt formula_cache formula with
-    | None ->
-        (* it should not actually be possible for a formula to
-           not be in the formula table
-
-           just don't cache it I guess
-        *)
-        Log.warn (fun m ->
-            m
-              "Tried to compute matches for a taint formula not in the cache \
-               (impossible?)");
-        compute_matches_fn ()
-    | Some (None, count) ->
-        let ranges, expls = compute_matches_fn () in
-        if count <= 1 then
-          (* if there's only 1 more use left, there's no point
-             in caching it
-          *)
-          (ranges, expls)
-        else (
-          (* otherwise, this is the first time we've seen this
-             formula, and we should cache it
-          *)
-          replace formula_cache formula (Some (ranges, expls), count - 1);
-          (ranges, expls))
-    | Some (Some (ranges, expls), count) ->
-        if count <= 1 then remove formula_cache formula;
-        (ranges, expls)
-end
-
-type formula_cache = ((RM.t list * ME.t list) option * int) Formula_tbl.t
-
-(* This function is for creating a formula cache which only caches formula that
-   it knows will be shared, at least once, among the formula in a bunch of
-   taint rules.
-
-   This is because it's obviously not useful to cache a formula's matches if
-   that formula never comes up again. This cache stores an option, with keys
-   that are only formula that are guaranteed to appear more than once in the
-   collection.
-*)
-let mk_specialized_formula_cache (rules : R.taint_rule list) =
-  let count_tbl = Formula_tbl.create 128 in
-  let flat_formulas =
-    rules
-    |> List.concat_map (fun (rule : R.taint_mode R.rule_info) ->
-           let (`Taint (spec : R.taint_spec)) = rule.R.mode in
-           R.formula_of_mode (`Taint spec))
-  in
-  flat_formulas
-  |> List.iter (fun formula ->
-         match Formula_tbl.find_opt count_tbl formula with
-         | None -> Formula_tbl.add count_tbl formula (None, 1)
-         | Some (_, x) -> Formula_tbl.replace count_tbl formula (None, 1 + x));
-  (* We return the table with pairs of (None, count) itself.
-     When we try to cache a find, we will first check whether decreasing this
-     counter results in 0. Then, there are no more uses, and the result is no
-     longer worth caching.
-     This way we don't keep around entries when we don't need to.
-  *)
-  count_tbl
-
-(*****************************************************************************)
-(* Finding matches for taint specs *)
-(*****************************************************************************)
-
-(* =~ List.concat_map with automatic management of matching-explanations *)
-let concat_map_with_expls f xs =
-  let all_expls = ref [] in
-  let res =
-    xs
-    |> List.concat_map (fun x ->
-           let ys, expls = f x in
-           Stack_.push expls all_expls;
-           ys)
-  in
-  (res, List_.flatten (List.rev !all_expls))
-
-let%test _ =
-  concat_map_with_expls (fun x -> ([ -x; x ], [ 2 * x; 3 * x ])) [ 0; 1; 2 ]
-  =*= ([ 0; 0; -1; 1; -2; 2 ], [ 0; 0; 2; 3; 4; 6 ])
-
-let find_range_w_metas formula_cache (xconf : Match_env.xconfig)
-    (xtarget : Xtarget.t) (rule : R.t) (specs : (R.formula * 'a) list) :
-    (RM.t * 'a) list * ME.t list =
-  (* TODO: Make an Or formula and run a single query. *)
-  (* if perf is a problem, we could build an interval set here *)
-  specs
-  |> concat_map_with_expls (fun (pf, x) ->
-         let ranges, expls =
-           Formula_tbl.cached_find_opt formula_cache pf (fun () ->
-               range_w_metas_of_formula xconf xtarget rule pf)
-         in
-         (ranges |> List_.map (fun rwm -> (rwm, x)), expls))
-
-let find_sanitizers_matches formula_cache (xconf : Match_env.xconfig)
-    (xtarget : Xtarget.t) (rule : R.t) (specs : R.taint_sanitizer list) :
-    (bool * RM.t * R.taint_sanitizer) list * ME.t list =
-  specs
-  |> concat_map_with_expls (fun (sanitizer : R.taint_sanitizer) ->
-         let ranges, expls =
-           Formula_tbl.cached_find_opt formula_cache sanitizer.sanitizer_formula
-             (fun () ->
-               range_w_metas_of_formula xconf xtarget rule
-                 sanitizer.sanitizer_formula)
-         in
-         ( ranges
-           |> List_.map (fun x -> (sanitizer.R.not_conflicting, x, sanitizer)),
-           expls ))
-
-(* Finds all matches of `pattern-propagators`. *)
-let find_propagators_matches formula_cache (xconf : Match_env.xconfig)
-    (xtarget : Xtarget.t) (rule : R.t)
-    (propagators_spec : R.taint_propagator list) =
-  propagators_spec
-  |> List.concat_map (fun (p : R.taint_propagator) ->
-         let mvar_pfrom, tok_pfrom = p.from in
-         let mvar_pto, tok_pto = p.to_ in
-         let ranges_w_metavars, _expsTODO =
-           Formula_tbl.cached_find_opt formula_cache p.propagator_formula
-             (fun () ->
-               range_w_metas_of_formula xconf xtarget rule p.propagator_formula)
-         in
-         (* Now, for each match of the propagator pattern, we try to construct
-          * a `propagator_match`. We just need to look up what code is captured
-          * by the metavariables `from` and `to`, and check if we can obtain good
-          * location info for that code (i.e., we have real tokens rather than
-          * fake ones). *)
-         ranges_w_metavars
-         |> List_.filter_map (fun rwm ->
-                (* The piece of code captured by the `from` metavariable.  *)
-                let* _mvar_from, mval_from =
-                  List.find_opt
-                    (fun (mvar, _mval) -> MV.equal_mvar mvar_pfrom mvar)
-                    rwm.RM.mvars
-                in
-                (* The piece of code captured by the `to` metavariable.  *)
-                let* _mvar_to, mval_to =
-                  List.find_opt
-                    (fun (mvar, _mval) -> MV.equal_mvar mvar_pto mvar)
-                    rwm.RM.mvars
-                in
-                (* TODO: log a warning when we cannot obtain a taint propagator due to
-                 * lacking range info. *)
-                match (Tok.loc_of_tok tok_pfrom, Tok.loc_of_tok tok_pto) with
-                | Error _, _
-                | _, Error _ ->
-                    None
-                | Ok loc_pfrom, Ok loc_pto ->
-                    let* mval_from_start_loc, mval_from_end_loc =
-                      AST_generic_helpers.range_of_any_opt
-                        (MV.mvalue_to_any mval_from)
-                    in
-                    let* mval_to_start_loc, mval_to_end_loc =
-                      AST_generic_helpers.range_of_any_opt
-                        (MV.mvalue_to_any mval_to)
-                    in
-                    let from =
-                      Range.range_of_token_locations mval_from_start_loc
-                        mval_from_end_loc
-                    in
-                    let to_ =
-                      Range.range_of_token_locations mval_to_start_loc
-                        mval_to_end_loc
-                    in
-                    let id =
-                      Common.spf "propagator:%d:%d:%d:%d:%d:%d"
-                        loc_pfrom.pos.bytepos loc_pto.pos.bytepos
-                        from.Range.start from.Range.end_ to_.Range.start
-                        to_.Range.end_
-                    in
-                    Some { id; rwm; from; to_; spec = p }))
-
 (*****************************************************************************)
 (* Testing whether some matches a taint spec *)
 (*****************************************************************************)
 
-let range_of_any any =
-  (* This is potentially slow. We may need to store range position in
-   * the AST at some point. *)
-  match AST_generic_helpers.range_of_any_opt any with
-  | None ->
-      (* IL.any_of_orig will return `G.Anys []` for `NoOrig`, and there is
-       * no point in issuing this warning in that case.
-       * TODO: Perhaps we should avoid the call to `any_in_ranges` in the
-       * first place? *)
-      if any <> G.Anys [] then
-        Log.debug (fun m ->
-            m "Cannot compute range, there are no real tokens in this AST: %s"
-              (G.show_any any));
-      None
-  | Some (tok1, tok2) ->
-      let r = Range.range_of_token_locations tok1 tok2 in
-      Some r
-
-(* Assuming that `r` is a subrange of `match_range` then this computes a
- * float in [0.0, 1.0]. We expect `r` to be the range of some arbitrary
- * piece of code,  and `match_range` to be the range of a match of some
- * taint spec (e.g. a taint source). Then this float indicates how much
- * overlap there is between the code and the spec. The degree of overlap
- * is used to determine whether the match is "exact" (overlap > 0.99),
- * which e.g. triggets the side-effectful propagation of taint. *)
-let overlap_with ~match_range r =
-  let r1 = match_range in
-  float_of_int (r.Range.end_ - r.Range.start + 1)
-  /. float_of_int (r1.Range.end_ - r1.Range.start + 1)
-
-let any_is_in_sources_matches rule any matches =
-  let ( let* ) = option_bind_list in
-  let* r = range_of_any any in
-  matches
-  |> List_.filter_map (fun (rwm, (ts : R.taint_source)) ->
-         if Range.( $<=$ ) r rwm.RM.r then
-           Some
-             (let spec_pm = RM.range_to_pattern_match_adjusted rule rwm in
-              let overlap = overlap_with ~match_range:rwm.RM.r r in
-              {
-                Taint_smatch.spec = ts;
-                spec_id = ts.source_id;
-                spec_pm;
-                range = r;
-                overlap;
-              })
-         else None)
-
-(* Builds a table to quickly look up for propagator matches, taking advantage of
- * the requirement that propagators must be exact matches.
- *
- * OBS: Previously we allowed these matches if they had an overlap of >0.99, but
- *   to be honest that was arbitrary and fragile (the overlap largely depends on
- *   the amount of text being matched). Typically propagators from/to match
- *   l-values and those typically we can count as being 100% perfect matches.
- *   If this causes problems we can always roll back, but all tests still work.
- *)
-let propagators_table_of_matches rule matches =
-  let mk_match prop var kind r =
-    let spec_pm = RM.range_to_pattern_match_adjusted rule prop.rwm in
-    let spec : D.a_propagator = { kind; prop = prop.spec; var } in
-    {
-      Taint_smatch.spec;
-      spec_id = prop.spec.propagator_id;
-      spec_pm;
-      range = r;
-      overlap = 1.0;
-    }
-  in
-  let tbl = Range_table.create () in
-  matches
-  |> List.iter (fun prop ->
-         let var = prop.id in
-         Range_table.push tbl prop.to_ (mk_match prop var `To prop.to_);
-         Range_table.push tbl prop.from (mk_match prop var `From prop.from));
-  tbl
-
-(* Check whether `any` matches either the `from` or the `to` of any of the
- * `pattern-propagators`. Matches must be exact to make taint propagation
- * more precise and predictable.
- *
- * THINK: Now that we have "Best_matches" we could perhaps use that for
- *   propagators too?
- *)
-let any_is_in_propagators_matches any tbl : D.a_propagator Taint_smatch.t list =
-  match range_of_any any with
-  | None -> []
-  | Some r -> Range_table.get tbl r
-
-let any_is_in_sanitizers_matches rule any matches =
-  let ( let* ) = option_bind_list in
-  let* r = range_of_any any in
-  matches
-  |> List_.filter_map (fun (rwm, spec) ->
-         if Range.( $<=$ ) r rwm.RM.r then
-           Some
-             (let spec_pm = RM.range_to_pattern_match_adjusted rule rwm in
-              let overlap = overlap_with ~match_range:rwm.RM.r r in
-              {
-                Taint_smatch.spec;
-                spec_id = spec.R.sanitizer_id;
-                spec_pm;
-                range = r;
-                overlap;
-              })
-         else None)
-
-let any_is_in_sinks_matches rule any matches =
-  let ( let* ) = option_bind_list in
-  let* r = range_of_any any in
-  matches
-  |> List_.filter_map (fun (rwm, (spec : R.taint_sink)) ->
-         if Range.( $<=$ ) r rwm.RM.r then
-           Some
-             (let spec_pm = RM.range_to_pattern_match_adjusted rule rwm in
-              let overlap = overlap_with ~match_range:rwm.RM.r r in
-              {
-                Taint_smatch.spec;
-                spec_id = spec.sink_id;
-                spec_pm;
-                range = r;
-                overlap;
-              })
-         else None)
-
 let lazy_force x = Lazy.force x [@@profiling]
 
 (*****************************************************************************)
@@ -633,129 +223,6 @@ let pms_of_finding ~match_on finding =
 (* Main entry points *)
 (*****************************************************************************)
 
-let taint_config_of_rule ~per_file_formula_cache xconf file ast_and_errors
-    ({ mode = `Taint spec; _ } as rule : R.taint_rule) handle_effects =
-  let file = Fpath.v file in
-  let formula_cache = per_file_formula_cache in
-  let xconf = Match_env.adjust_xconfig_with_rule_options xconf rule.options in
-  let lazy_ast_and_errors = lazy ast_and_errors in
-  (* TODO: should this function just take a target, rather than a file? *)
-  let xtarget : Xtarget.t =
-    {
-      path = { origin = File file; internal_path_to_content = file };
-      xlang = rule.target_analyzer;
-      lazy_content = lazy (UFile.read_file file);
-      lazy_ast_and_errors;
-    }
-  in
-  let (sources_ranges : (RM.t * R.taint_source) list), expls_sources =
-    find_range_w_metas formula_cache xconf xtarget rule
-      (spec.sources |> snd
-      |> List_.map (fun (src : R.taint_source) -> (src.source_formula, src)))
-  and (propagators_ranges : propagator_match list) =
-    find_propagators_matches formula_cache xconf xtarget rule spec.propagators
-  and (sinks_ranges : (RM.t * R.taint_sink) list), expls_sinks =
-    find_range_w_metas formula_cache xconf xtarget rule
-      (spec.sinks |> snd
-      |> List_.map (fun (sink : R.taint_sink) -> (sink.sink_formula, sink)))
-  in
-  let sanitizers_ranges, expls_sanitizers =
-    match spec.sanitizers with
-    | None -> ([], [])
-    | Some (_, sanitizers_spec) ->
-        find_sanitizers_matches formula_cache xconf xtarget rule sanitizers_spec
-  in
-  let (sanitizers_ranges : (RM.t * R.taint_sanitizer) list) =
-    (* A sanitizer cannot conflict with a sink or a source, otherwise it is
-     * filtered out. This allows to e.g. declare `$F(...)` as a sanitizer,
-     * to assume that any other function will handle tainted data safely.
-     * Without this, `$F(...)` will automatically sanitize any other function
-     * call acting as a sink or a source. *)
-    sanitizers_ranges
-    |> List_.filter_map (fun (not_conflicting, range, spec) ->
-           (* TODO: Warn user when we filter out a sanitizer? *)
-           if not_conflicting then
-             if
-               not
-                 (List.exists
-                    (fun (range', _) -> range'.RM.r =*= range.RM.r)
-                    sinks_ranges
-                 || List.exists
-                      (fun (range', _) -> range'.RM.r =*= range.RM.r)
-                      sources_ranges)
-             then Some (range, spec)
-             else None
-           else Some (range, spec))
-  in
-  let expls =
-    if xconf.matching_explanations then
-      let ranges_to_pms ranges_and_stuff =
-        ranges_and_stuff
-        |> List_.map (fun (rwm, _) ->
-               RM.range_to_pattern_match_adjusted rule rwm)
-      in
-      [
-        {
-          ME.op = OutJ.TaintSource;
-          pos = fst spec.sources;
-          children = expls_sources;
-          matches = ranges_to_pms sources_ranges;
-          extra = None;
-        };
-        {
-          ME.op = OutJ.TaintSink;
-          pos = fst spec.sinks;
-          children = expls_sinks;
-          matches = ranges_to_pms sinks_ranges;
-          extra = None;
-        }
-        (* TODO: propagators *);
-      ]
-      @
-      match spec.sanitizers with
-      | None -> []
-      | Some (tok, _) ->
-          [
-            {
-              ME.op = OutJ.TaintSanitizer;
-              pos = tok;
-              children = expls_sanitizers;
-              (* 'sanitizer_ranges' will be affected by `not-conflicting: true`:
-               * if a sanitizer coincides exactly with a source/sink then it will
-               * be filtered out. So the sanitizer matches may not be the union of
-               * the matches of the individual sanitizers. Anyhow, not-conflicting
-               * has been deprecated for quite some time, and we will remove it at
-               * some point. *)
-              matches = ranges_to_pms sanitizers_ranges;
-              extra = None;
-            };
-          ]
-    else []
-  in
-  let propagators_tbl = propagators_table_of_matches rule propagators_ranges in
-  let config = xconf.config in
-  ( {
-      Dataflow_tainting.filepath = !!file;
-      rule_id = fst rule.R.id;
-      track_control =
-        spec.sources |> snd
-        |> List.exists (fun (src : R.taint_source) -> src.source_control);
-      is_source = (fun x -> any_is_in_sources_matches rule x sources_ranges);
-      is_propagator = (fun x -> any_is_in_propagators_matches x propagators_tbl);
-      is_sanitizer =
-        (fun x -> any_is_in_sanitizers_matches rule x sanitizers_ranges);
-      is_sink = (fun x -> any_is_in_sinks_matches rule x sinks_ranges);
-      unify_mvars = config.taint_unify_mvars;
-      handle_effects;
-    },
-    {
-      sources = sources_ranges;
-      sanitizers = sanitizers_ranges |> List_.map fst;
-      sinks = sinks_ranges;
-    },
-    expls )
-[@@trace_trace]
-
 let check_var_def lang options taint_config env id ii expr =
   let name = AST_to_IL.var_of_id_info id ii in
   let assign =
@@ -778,7 +245,7 @@ let add_to_env lang options taint_config env id ii opt_expr =
   let var_type = Typing.resolved_type_of_id_info lang var.id_info in
   let id_taints =
     taint_config.D.is_source (G.Tk (snd id))
-    |> List_.map (fun (x : _ Taint_smatch.t) -> (x.spec_pm, x.spec))
+    |> List_.map (fun (x : _ Taint_spec_match.t) -> (x.spec_pm, x.spec))
     (* These sources come from the parameters to a function,
         which are not within the normal control flow of a code.
         We can safely say there's no incoming taints to these sources.
@@ -949,13 +416,13 @@ let check_rule per_file_formula_cache (rule : R.taint_rule) match_hook
           `Source
       | `Sink, `Sink -> `Sink
     in
-    let handle_effects _ effects _env =
+    let handle_effects _fun_name effects =
       effects
       |> List.iter (fun effect ->
              pms_of_finding ~match_on effect
              |> List.iter (fun pm -> Stack_.push pm matches))
     in
-    taint_config_of_rule ~per_file_formula_cache xconf
+    Match_taint_spec.taint_config_of_rule ~per_file_formula_cache xconf
       !!internal_path_to_content (ast, []) rule handle_effects
   in
 
@@ -1059,7 +526,9 @@ let check_rules ~match_hook
      In particular, this expects to see big gains due to shared propagators,
      in Semgrep Pro. There may be some benefit in OSS, but it's low-probability.
   *)
-  let per_file_formula_cache = mk_specialized_formula_cache rules in
+  let per_file_formula_cache =
+    Formula_cache.mk_specialized_formula_cache rules
+  in
 
   rules
   |> List_.map (fun rule ->
diff --git a/src/engine/Match_tainting_mode.mli b/src/engine/Match_tainting_mode.mli
index c24128efa125..6b43b672c3bb 100644
--- a/src/engine/Match_tainting_mode.mli
+++ b/src/engine/Match_tainting_mode.mli
@@ -1,23 +1,3 @@
-type debug_taint = {
-  sources : (Range_with_metavars.t * Rule.taint_source) list;
-      (** Ranges matched by `pattern-sources:` *)
-  sanitizers : Range_with_metavars.ranges;
-      (** Ranges matched by `pattern-sanitizers:` *)
-  sinks : (Range_with_metavars.t * Rule.taint_sink) list;
-      (** Ranges matched by `pattern-sinks:` *)
-}
-(** To facilitate debugging of taint rules. *)
-
-(* The type of the specialized formual cache used for inter-rule
-   match sharing.
-*)
-type formula_cache
-
-(* These formula caches are only safe to use to share results between
-   runs of rules on the same target! It is consumed by [taint_config_of_rule].
-*)
-val mk_specialized_formula_cache : Rule.taint_rule list -> formula_cache
-
 val hook_setup_hook_function_taint_signature :
   (Match_env.xconfig ->
   Rule.taint_rule ->
@@ -40,24 +20,6 @@ val hook_setup_hook_function_taint_signature :
   *   (or we could infer a signature for them too...).
   *)
 
-(* It could be a private function, but it is also used by Deep Semgrep. *)
-(* This [formula_cache] argument is exposed here because this function is also
-   a subroutine but the cache itself should be created outside of the any main
-   loop which runs over rules. This cache is only safe to share with if
-   [taint_config_of_rule] is used on the same file!
-*)
-val taint_config_of_rule :
-  per_file_formula_cache:formula_cache ->
-  Match_env.xconfig ->
-  string (* filename *) ->
-  AST_generic.program * Tok.location list ->
-  Rule.taint_rule ->
-  (Dataflow_tainting.var option ->
-  Shape_and_sig.Effect.t list ->
-  Taint_lval_env.t ->
-  unit) ->
-  Dataflow_tainting.config * debug_taint * Matching_explanation.t list
-
 val mk_fun_input_env :
   Language.t ->
   Rule_options_t.t ->
diff --git a/src/matching/Generic_vs_generic.ml b/src/matching/Generic_vs_generic.ml
index 8f4ec2e69011..8c35e45ebded 100644
--- a/src/matching/Generic_vs_generic.ml
+++ b/src/matching/Generic_vs_generic.ml
@@ -943,6 +943,15 @@ and m_expr ?(is_root = false) ?(arguments_have_changed = true) a b =
    *)
   | G.Ellipsis _a1, _ -> return ()
   | G.DeepEllipsis (_, a1, _), _b -> m_expr_deep a1 b
+  (* equivalence: (-) E vs -n ==> E vs n *)
+  | ( G.Call ({ e = G.IdSpecial (G.Op G.Minus, _); _ }, (_, [ G.Arg arga ], _)),
+      B.L (B.Int int_lit) ) ->
+      m_expr arga (B.L (B.Int (Parsed_int.neg int_lit)) |> B.e)
+  (* equivalence: -n vs (-) E ==> n vs E *)
+  | ( G.L (G.Int int_lit),
+      B.Call ({ e = B.IdSpecial (B.Op B.Minus, _); _ }, (_, [ B.Arg argb ], _))
+    ) ->
+      m_expr (G.L (G.Int (Parsed_int.neg int_lit)) |> G.e) argb
   (* must be before constant propagation case below *)
   | G.L a1, B.L b1 -> m_literal a1 b1
   (* equivalence: constant propagation and evaluation!
@@ -1878,7 +1887,7 @@ and m_call_op aop toka aargs bop tokb bargs tin =
             Log.warn (fun m ->
                 m
                   "Will not perform AC-matching, something went wrong when \
-                   trying to convert operands to AC normal form: %s ~ %s"
+                   trying to convert operands to AC normal form: %s vs %s"
                   (G.show_expr
                      (G.Call (G.IdSpecial (G.Op aop, toka) |> G.e, aargs) |> G.e))
                   (B.show_expr
@@ -2717,6 +2726,7 @@ and m_stmt a b =
   | G.OtherStmtWithStmt (a1, a2, a3), B.OtherStmtWithStmt (b1, b2, b3) ->
       m_other_stmt_with_stmt_operator a1 b1 >>= fun () ->
       m_list m_any a2 b2 >>= fun () -> m_stmt a3 b3
+  | G.RawStmt a, B.RawStmt b -> m_raw_tree a b
   | G.WithUsingResource (a1, a2, a3), B.WithUsingResource (b1, b2, b3) ->
       m_tok a1 b1 >>= fun () ->
       m_list m_stmt a2 b2 >>= fun () -> m_stmt a3 b3
@@ -2739,6 +2749,7 @@ and m_stmt a b =
   | G.Assert _, _
   | G.OtherStmt _, _
   | G.OtherStmtWithStmt _, _
+  | G.RawStmt _, _
   | G.WithUsingResource _, _ ->
       fail ()
 
diff --git a/src/matching/SubAST_generic.ml b/src/matching/SubAST_generic.ml
index 106e42753383..2e080b558eff 100644
--- a/src/matching/SubAST_generic.ml
+++ b/src/matching/SubAST_generic.ml
@@ -36,6 +36,14 @@ let subexprs_of_any_list xs =
          | _ -> x)
        []
 
+let substmts_of_any_list xs =
+  xs
+  |> List.fold_left
+       (fun x -> function
+         | S s -> s :: x
+         | _ -> x)
+       []
+
 (* used for really deep statement matching *)
 
 let subexprs_of_stmt_kind = function
@@ -79,6 +87,7 @@ let subexprs_of_stmt_kind = function
            | _ -> None)
   | OtherStmt (_op, xs) -> subexprs_of_any_list xs
   | OtherStmtWithStmt (_, xs, _) -> subexprs_of_any_list xs
+  | RawStmt x -> Raw_tree.anys x |> subexprs_of_any_list
   (* 0 *)
   | DirectiveStmt _
   | Block _
@@ -332,6 +341,7 @@ let substmts_of_stmt st =
         | FuncDef def -> [ H.funcbody_to_stmt def.fbody ]
         | ClassDef def ->
             def.cbody |> Tok.unbracket |> List_.map (function F st -> st))
+  | RawStmt x -> Raw_tree.anys x |> substmts_of_any_list
 
 (*****************************************************************************)
 (* Visitors  *)
diff --git a/src/metachecking/Translate_rule.ml b/src/metachecking/Translate_rule.ml
index 65904b87235f..b9d2c82c4e4d 100644
--- a/src/metachecking/Translate_rule.ml
+++ b/src/metachecking/Translate_rule.ml
@@ -68,15 +68,17 @@ let rec translate_metavar_cond cond : [> `O of (string * Yaml.value) list ] =
         match lang with
         | None -> []
         | Some x -> [ ("language", `String (Xlang.to_string x)) ])
-  | CondName { mvar; kind; module_ } ->
+  | CondName { mvar; kind; modules } ->
       `O
         ([ ("metavariable", `String mvar) ]
         @ (match kind with
           | Some DjangoView -> [ ("kind", `String "django-view") ]
           | None -> [])
         @
-        match module_ with
-        | Some name -> [ ("module", `String name) ]
+        match modules with
+        | Some [ name ] -> [ ("module", `String name) ]
+        | Some names ->
+            [ ("modules", `A (List_.map (fun n -> `String n) names)) ]
         | None -> [])
   | CondRegexp (mv, re_str, _) ->
       `O [ ("metavariable", `String mv); ("regex", `String re_str) ]
diff --git a/src/osemgrep/cli/Help.ml b/src/osemgrep/cli/Help.ml
index 1d57cd6958f0..a837aeaa525a 100644
--- a/src/osemgrep/cli/Help.ml
+++ b/src/osemgrep/cli/Help.ml
@@ -59,7 +59,7 @@ For more information about Semgrep, visit @{<cyan;ul>https://semgrep.dev@}
 @{<ul>Help@}:
   @{<cyan>semgrep COMMAND --help@}       For more information on each command
 
-For the CLI docs visit @{<cyan;ul>https://semgrep.dev/docs/category/semgrep-cli/@}
+For the CLI docs visit @{<cyan;ul>https://semgrep.dev/docs/cli-reference@}
 |}
 
 let print_semgrep_dashdash_help (stdout : Cap.Console.stdout) =
diff --git a/src/osemgrep/cli_ci/Ci_CLI.ml b/src/osemgrep/cli_ci/Ci_CLI.ml
index d557d6d62875..c8bc22ec3c8f 100644
--- a/src/osemgrep/cli_ci/Ci_CLI.ml
+++ b/src/osemgrep/cli_ci/Ci_CLI.ml
@@ -138,6 +138,10 @@ let o_config : string list Term.t =
   in
   Arg.value (Arg.opt_all Arg.string [] info)
 
+let o_x_dump_n_rule_partitions : int Term.t =
+  let info = Arg.info [ "x-dump-rule-partitions" ] ~doc:{|Internal flag.|} in
+  Arg.value (Arg.opt Arg.int 0 info)
+
 (*************************************************************************)
 (* 'scan' subset supported by 'ci' *)
 (*************************************************************************)
@@ -313,7 +317,8 @@ let cmdline_term : conf Term.t =
    * variables (Romain's idea).
    *)
   let combine scan_conf audit_on code secrets dry_run _internal_ci_scan_results
-      subdir supply_chain suppress_errors _git_meta _github_meta =
+      _x_dump_n_rule_partitions subdir supply_chain suppress_errors _git_meta
+      _github_meta =
     let products =
       (if secrets then [ `Secrets ] else [])
       @ (if code then [ `SAST ] else [])
@@ -323,9 +328,9 @@ let cmdline_term : conf Term.t =
   in
   Term.(
     const combine $ scan_subset_cmdline_term $ o_audit_on $ o_code
-    $ SC.o_secrets $ o_dry_run $ o_internal_ci_scan_results $ o_subdir
-    $ o_supply_chain $ o_suppress_errors $ Git_metadata.env
-    $ Github_metadata.env)
+    $ SC.o_secrets $ o_dry_run $ o_internal_ci_scan_results
+    $ o_x_dump_n_rule_partitions $ o_subdir $ o_supply_chain $ o_suppress_errors
+    $ Git_metadata.env $ Github_metadata.env)
 
 let doc = "the recommended way to run semgrep in CI"
 
diff --git a/src/osemgrep/cli_ci/Ci_subcommand.ml b/src/osemgrep/cli_ci/Ci_subcommand.ml
index b01b2b2e005d..e457c3092307 100644
--- a/src/osemgrep/cli_ci/Ci_subcommand.ml
+++ b/src/osemgrep/cli_ci/Ci_subcommand.ml
@@ -201,7 +201,7 @@ let scan_config_and_rules_from_deployment ~dry_run
   let scan_metadata : OutJ.scan_metadata =
     {
       cli_version = Version.version;
-      unique_id = Uuidm.v `V4;
+      unique_id = Uuidm.v4_gen (Stdlib.Random.State.make_self_init ()) ();
       (* TODO: should look at conf.secrets, conf.sca, conf.code, etc. *)
       requested_products = [];
       dry_run = false;
diff --git a/src/osemgrep/cli_scan/Scan_CLI.ml b/src/osemgrep/cli_scan/Scan_CLI.ml
index c83cdacdb385..f5e66ceeb4ba 100644
--- a/src/osemgrep/cli_scan/Scan_CLI.ml
+++ b/src/osemgrep/cli_scan/Scan_CLI.ml
@@ -641,7 +641,8 @@ This may still run Pro rules, but only using the OSS features.
 let o_pro_languages : bool Term.t =
   let info =
     Arg.info [ "pro-languages" ]
-      ~doc:("Enable Pro languages (currently Apex and Elixir). " ^ C.blurb_pro)
+      ~doc:
+        ("Enable Pro languages (currently Apex, C#, and Elixir). " ^ C.blurb_pro)
   in
   Arg.value (Arg.flag info)
 
@@ -665,8 +666,8 @@ let o_pro : bool Term.t =
   let info =
     Arg.info [ "pro" ]
       ~doc:
-        ("Inter-file analysis and Pro languages (currently Apex and Elixir). "
-       ^ C.blurb_pro)
+        ("Inter-file analysis and Pro languages (currently Apex, C#, and \
+          Elixir. " ^ C.blurb_pro)
   in
   Arg.value (Arg.flag info)
 
diff --git a/src/osemgrep/cli_validate/Validate_subcommand.ml b/src/osemgrep/cli_validate/Validate_subcommand.ml
index 511269aef326..be4bae1bd43c 100644
--- a/src/osemgrep/cli_validate/Validate_subcommand.ml
+++ b/src/osemgrep/cli_validate/Validate_subcommand.ml
@@ -29,9 +29,14 @@ module Out = Semgrep_output_v1_t
  *
  *  (2) Semgrep rule schema errors (e.g., missing 'message:' field). We
  *    detect those thanks to Parse_rule.ml
+ *    Note that Parse_rule.ml also parses the patterns and validates they
+ *    have the right syntax (unlike jsonschema). This is why
+ *    osemgrep validate accepts a --pro flag as opposed to pysemgrep because
+ *    we need access to the pro languages parsers to parse the pro languages
+ *    patterns.
  *    TODO: use the OCaml grace library of Cooper for better error messaging?
- *    or fallback to jsonschema in pysemgrep just for the error message like
- *    zz did?
+ *     or fallback to jsonschema in pysemgrep just for the error message like
+ *     zz did?
  *
  *  (3) "Logical" errors, by running Semgrep rules (also called "meta" rules)
  *     on those target rules. We run Semgrep on Semgrep! We detect
diff --git a/src/osemgrep/configuring/Semgrep_settings.ml b/src/osemgrep/configuring/Semgrep_settings.ml
index 348cfd02627b..90928fbfb6ec 100644
--- a/src/osemgrep/configuring/Semgrep_settings.ml
+++ b/src/osemgrep/configuring/Semgrep_settings.ml
@@ -20,10 +20,11 @@ type t = {
 }
 
 let default =
+  let rand = Stdlib.Random.State.make_self_init () in
   {
     has_shown_metrics_notification = None;
     api_token = None;
-    anonymous_user_id = Uuidm.v `V4;
+    anonymous_user_id = Uuidm.v4_gen rand ();
   }
 
 (*****************************************************************************)
diff --git a/src/osemgrep/core/Metrics_.ml b/src/osemgrep/core/Metrics_.ml
index d3bff434a6e2..3f731977b453 100644
--- a/src/osemgrep/core/Metrics_.ml
+++ b/src/osemgrep/core/Metrics_.ml
@@ -144,8 +144,9 @@ type t = {
 let now () : Timedesc.Timestamp.t = Timedesc.Timestamp.now ()
 
 let default_payload =
+  let rand = Stdlib.Random.State.make_self_init () in
   {
-    Semgrep_metrics_t.event_id = Uuidm.v `V4;
+    Semgrep_metrics_t.event_id = Uuidm.v4_gen rand ();
     anonymous_user_id = "";
     started_at = now ();
     sent_at = now ();
diff --git a/src/osemgrep/language_server/Unit_LS.ml b/src/osemgrep/language_server/Unit_LS.ml
index 6140852ed981..9adf9083cf3f 100644
--- a/src/osemgrep/language_server/Unit_LS.ml
+++ b/src/osemgrep/language_server/Unit_LS.ml
@@ -103,7 +103,7 @@ let mock_run_results (files : string list) : Core_runner.result =
 
 let mock_workspace ?(git = false) () : Fpath.t =
   let rand_dir () =
-    let uuid = Uuidm.v `V4 in
+    let uuid = Uuidm.v4_gen (Stdlib.Random.State.make_self_init ()) () in
     let dir_name = "test_workspace_" ^ Uuidm.to_string uuid in
     let dir = Filename.concat (Filename.get_temp_dir_name ()) dir_name in
     Unix.mkdir dir 0o777;
diff --git a/src/osemgrep/language_server/server/Lsp_.ml b/src/osemgrep/language_server/server/Lsp_.ml
index 15d505933268..344f10a40b61 100644
--- a/src/osemgrep/language_server/server/Lsp_.ml
+++ b/src/osemgrep/language_server/server/Lsp_.ml
@@ -82,7 +82,9 @@ let respond (type r) (id : Id.t) (request : r CR.t) (response : r) =
 
 (** Send a request to the client *)
 let request request =
-  let id = Uuidm.v `V4 |> Uuidm.to_string in
+  let id =
+    Uuidm.v4_gen (Stdlib.Random.State.make_self_init ()) () |> Uuidm.to_string
+  in
   let request = SR.to_jsonrpc_request request (`String id) in
   Logs.debug (fun m ->
       m "Sending request %s"
@@ -117,7 +119,9 @@ let notify_show_message ~kind s =
 
 (** Show a little progress circle while doing thing. Returns a token needed to end progress*)
 let create_progress title message =
-  let id = Uuidm.v `V4 |> Uuidm.to_string in
+  let id =
+    Uuidm.v4_gen (Stdlib.Random.State.make_self_init ()) () |> Uuidm.to_string
+  in
   Logs.debug (fun m ->
       m "Creating progress token %s, (%s: %s)" id title message);
   let token = ProgressToken.t_of_yojson (`String id) in
diff --git a/src/osemgrep/networking/Semgrep_login.ml b/src/osemgrep/networking/Semgrep_login.ml
index 3675983d348f..4d98a4c8f673 100644
--- a/src/osemgrep/networking/Semgrep_login.ml
+++ b/src/osemgrep/networking/Semgrep_login.ml
@@ -27,7 +27,8 @@ type login_session = shared_secret * Uri.t
 let support_url = "https://semgrep.dev/docs/support/"
 
 let make_login_url () =
-  let session_id = Uuidm.v `V4 in
+  let rand = Stdlib.Random.State.make_self_init () in
+  let session_id = Uuidm.v4_gen rand () in
   ( session_id,
     Uri.(
       add_query_params'
diff --git a/src/osemgrep/reporting/Gitlab_output.ml b/src/osemgrep/reporting/Gitlab_output.ml
index 189085988013..5f65cdaa8ff4 100644
--- a/src/osemgrep/reporting/Gitlab_output.ml
+++ b/src/osemgrep/reporting/Gitlab_output.ml
@@ -94,7 +94,7 @@ let format_cli_match (cli_match : OutT.cli_match) =
   let id =
     (* TODO the ?index argument needs to be provided (for ci_unique_key duplicates) *)
     Semgrep_hashing_functions.ci_unique_key cli_match
-    |> Uuidm.of_bytes |> Option.get |> Uuidm.to_string
+    |> Uuidm.of_binary_string |> Option.get |> Uuidm.to_string
   in
   let r =
     [
diff --git a/src/parsing/Check_pattern.ml b/src/parsing/Check_pattern.ml
index 31b8308b4184..3abe0876606e 100644
--- a/src/parsing/Check_pattern.ml
+++ b/src/parsing/Check_pattern.ml
@@ -46,7 +46,8 @@ let lang_has_no_dollar_ids =
     | Cairo
     | Scala
     | Solidity ->
-        false)
+        false
+    | Move_on_sui -> failwith "Unimplemented")
 
 class ['self] metavar_checker =
   object (_self : 'self)
diff --git a/src/parsing/Parse_rule_formula.ml b/src/parsing/Parse_rule_formula.ml
index d5652c21f42c..76e47745a824 100644
--- a/src/parsing/Parse_rule_formula.ml
+++ b/src/parsing/Parse_rule_formula.ml
@@ -236,7 +236,7 @@ type extra =
   | MetavarPattern of MV.mvar * Xlang.t option * Rule.formula
   | MetavarComparison of metavariable_comparison
   | MetavarAnalysis of MV.mvar * Rule.metavar_analysis_kind
-  | MetavarName of MV.mvar * Rule.metavar_name_kind option * string option
+  | MetavarName of MV.mvar * Rule.metavar_name_kind option * string list option
 
 (* old: | PatWherePython of string, but it was too dangerous.
  * MetavarComparison is not as powerful, but safer.
@@ -413,8 +413,8 @@ and parse_pair_old env ((key, value) : key * G.expr) :
                     | Some true -> rewrite_metavar_comparison_strip comparison
                     | _ -> comparison)
               | MetavarAnalysis (mvar, kind) -> R.CondAnalysis (mvar, kind)
-              | MetavarName (mvar, kind, module_) ->
-                  R.CondName { mvar; kind; module_ }
+              | MetavarName (mvar, kind, modules) ->
+                  R.CondName { mvar; kind; modules }
             in
             match
               ( H.dict_take_opt dict "focus-metavariable",
@@ -654,9 +654,16 @@ and parse_extra (env : env) (key : key) (value : G.expr) :
       in
       let/ kind_str = take_opt mv_name_dict env parse_string_wrap "kind" in
       let/ module_ = take_opt mv_name_dict env parse_string "module" in
-      match (kind_str, module_) with
-      | None, None ->
-          error_at_key env.id key "expected at least one of kind, module"
+      let/ modules =
+        take_opt mv_name_dict env
+          (fun e k -> parse_list e k (fun e -> parse_string e k))
+          "modules"
+      in
+      match (kind_str, module_, modules) with
+      | None, None, None ->
+          error_at_key env.id key "expected at least one of kind, module(s)"
+      | _, Some _, Some _ ->
+          error_at_key env.id key "expected only one of module, modules"
       | _ ->
           let/ kind =
             match Option.map parse_kind kind_str with
@@ -664,7 +671,13 @@ and parse_extra (env : env) (key : key) (value : G.expr) :
             | Some (Ok x) -> Ok (Some x)
             | Some (Error e) -> Error e
           in
-          Ok (MetavarName (mvar, kind, module_)))
+          Ok
+            (MetavarName
+               ( mvar,
+                 kind,
+                 match modules with
+                 | None -> Option.map (fun x -> [ x ]) module_
+                 | _ -> modules )))
   | _ -> error_at_key env.id key ("wrong parse_extra field: " ^ fst key)
 
 (*****************************************************************************)
diff --git a/src/parsing/Parsing_plugin.ml b/src/parsing/Parsing_plugin.ml
index 582b95d9ab08..5f78be2a7ca9 100644
--- a/src/parsing/Parsing_plugin.ml
+++ b/src/parsing/Parsing_plugin.ml
@@ -33,6 +33,8 @@ type target_file_parser =
   Fpath.t -> (AST_generic.program, unit) Tree_sitter_run.Parsing_result.t
 
 module type T = sig
+  val is_optional : bool
+
   val register_parsers :
     parse_pattern:pattern_parser -> parse_target:target_file_parser -> unit
 
@@ -43,6 +45,7 @@ end
 
 exception Missing_plugin of string
 
+(* Table of missing plugins that are not optional *)
 let missing_plugins : (Lang.t, unit) Hashtbl.t = Hashtbl.create 10
 
 (*****************************************************************************)
@@ -52,7 +55,7 @@ let missing_plugins : (Lang.t, unit) Hashtbl.t = Hashtbl.create 10
 let missing_plugin_msg lang =
   spf
     "Missing Semgrep extension needed for parsing %s target. Try adding \
-     `--pro` to your command."
+     `--pro-languages` to your command."
     (Lang.to_string lang)
 
 let check_if_missing lang =
@@ -78,10 +81,13 @@ let check_if_missing_analyzer (analyzer : Xlang.t) =
           | Some msg -> Error msg)
       | Error _ as res -> res)
 
+let all_possible_plugins = ref []
+
 (* Create and manage the reference holding a plugin. *)
-let make lang =
+let make ?(optional = false) lang =
+  all_possible_plugins := lang :: !all_possible_plugins;
   let parsers = ref None in
-  Hashtbl.add missing_plugins lang ();
+  if not optional then Hashtbl.add missing_plugins lang ();
   let register ~parse_pattern ~parse_target =
     match !parsers with
     | None ->
@@ -119,18 +125,27 @@ let make lang =
         raise (Missing_plugin msg)
     | Some (_, parse_target) -> parse_target file
   in
-  (register, is_available, parse_pattern, parse_target)
+  (optional, register, is_available, parse_pattern, parse_target)
 
 (*****************************************************************************)
 (* Plugins *)
 (*****************************************************************************)
 
 module Apex = struct
-  let register_parsers, is_available, parse_pattern, parse_target =
+  let is_optional, register_parsers, is_available, parse_pattern, parse_target =
     make Lang.Apex
 end
 
+(* Parsing Csharp can be done with the default open-source parser which
+   is not as good as this one. *)
+module Csharp = struct
+  let is_optional, register_parsers, is_available, parse_pattern, parse_target =
+    make ~optional:true Lang.Csharp
+end
+
 module Elixir = struct
-  let register_parsers, is_available, parse_pattern, parse_target =
+  let is_optional, register_parsers, is_available, parse_pattern, parse_target =
     make Lang.Elixir
 end
+
+let all_possible_plugins = List.rev !all_possible_plugins
diff --git a/src/parsing/Parsing_plugin.mli b/src/parsing/Parsing_plugin.mli
index 8deafbd3e996..cb858a642820 100644
--- a/src/parsing/Parsing_plugin.mli
+++ b/src/parsing/Parsing_plugin.mli
@@ -10,6 +10,10 @@ type target_file_parser =
   Fpath.t -> (AST_generic.program, unit) Tree_sitter_run.Parsing_result.t
 
 module type T = sig
+  (* If true, the plugin provides only an alternate implementation to
+     one that is available by default. *)
+  val is_optional : bool
+
   (* Register parsing functions for a language (called in semgrep-pro) *)
   val register_parsers :
     parse_pattern:pattern_parser -> parse_target:target_file_parser -> unit
@@ -28,18 +32,21 @@ end
  * and raise Missing_plugin when the plugin is not registered.
  *)
 module Apex : T
+module Csharp : T
 module Elixir : T
 
 (* Exception indicating that a plugin is missing. The argument is
    a human-readable error message. *)
 exception Missing_plugin of string
 
-(* Return an error message in case of a missing plugin. This function is
- * called in Parse_rule.ml to filter rules using languages that can not
- * be handled because their corresponding plugins was not registered.
- *)
+(* Return an error message in case of a missing plugin that is not optional.
+   This function is called in Parse_rule.ml to filter rules using languages
+   that can not be handled because their corresponding plugins was not
+   registered.
+*)
 val check_if_missing : Lang.t -> (unit, string) Result.t
 
 (* Call 'check_is_missing' if any target programming language with a missing
    plugin is involved with this analyzer. *)
 val check_if_missing_analyzer : Xlang.t -> (unit, string) Result.t
+val all_possible_plugins : Lang.t list
diff --git a/src/parsing/Unit_parsing.ml b/src/parsing/Unit_parsing.ml
index f6131b8a97f7..f247bc86d51b 100644
--- a/src/parsing/Unit_parsing.ml
+++ b/src/parsing/Unit_parsing.ml
@@ -16,21 +16,24 @@ let t = Testo.create
 (* ran from the root of the semgrep repository *)
 let tests_path = Fpath.v "tests"
 let tests_path_parsing = tests_path / "parsing"
+let tests_path_parsing_missing = tests_path / "parsing_missing"
+let tests_path_parsing_partial = tests_path / "parsing_partial"
+let tests_path_parsing_todo = tests_path / "parsing_todo"
 
 (*****************************************************************************)
 (* Helpers *)
 (*****************************************************************************)
 
-type error_tolerance = Strict | Missing_tokens | Partial_parsing
+type error_tolerance = Strict | Missing_tokens | Partial_parsing | Todo
 
 (*
    Strict parsing: errors due to missing (inserted) tokens are not tolerated
    in these tests.
 *)
-let parsing_tests_for_lang files lang =
+let parsing_tests_for_lang ?expected_outcome files lang =
   files
   |> List_.map (fun file ->
-         Testo.create ~tags:(Test_tags.tags_of_lang lang)
+         Testo.create ?expected_outcome ~tags:(Test_tags.tags_of_lang lang)
            (Filename.basename file) (fun () ->
              Parse_target.parse_and_resolve_name_strict lang (Fpath.v file)
              |> ignore))
@@ -82,6 +85,8 @@ let parsing_tests_for_lang error_tolerance files lang =
   | Strict -> parsing_tests_for_lang files lang
   | Missing_tokens -> missing_tokens_tests_for_lang files lang
   | Partial_parsing -> partial_parsing_tests_for_lang files lang
+  | Todo ->
+      parsing_tests_for_lang ~expected_outcome:(Should_fail "to do") files lang
 
 (*****************************************************************************)
 (* Tests *)
@@ -90,15 +95,15 @@ let parsing_tests_for_lang error_tolerance files lang =
 let pack_parsing_tests_for_lang ?(error_tolerance = Strict) lang =
   let slang = Lang.show lang in
   let dir = Lang.to_lowercase_alnum lang in
-  let dir = tests_path_parsing / dir in
   let exts = Lang.ext_of_lang lang in
   let dir, subcategory =
     match error_tolerance with
-    | Strict -> (dir, None)
-    | Missing_tokens -> (dir / "parsing_missing", Some "missing tokens")
-    | Partial_parsing -> (dir / "parsing_partial", Some "partial parsing")
+    | Strict -> (tests_path_parsing / dir, None)
+    | Missing_tokens -> (tests_path_parsing_missing / dir, Some "missing tokens")
+    | Partial_parsing ->
+        (tests_path_parsing_partial / dir, Some "partial parsing")
+    | Todo -> (tests_path_parsing_todo / dir, None)
   in
-  let is_dir f = f = "parsing_missing" || f = "parsing_partial" in
   let check_ext file =
     if
       not
@@ -108,13 +113,13 @@ let pack_parsing_tests_for_lang ?(error_tolerance = Strict) lang =
              file =~ regex)
            exts)
     then
-      failwith (spf "Unrecognized extention for file %s for lang %s" file slang)
-  in
-  (* Get all files then check extentions *)
-  let files =
-    Common2.glob (spf "%s/*" !!dir) |> List.filter (fun f -> not (is_dir f))
+      failwith (spf "Unrecognized extension for file %s for lang %s" file slang)
   in
-  if files =*= [] then failwith (spf "Empty set of parsing tests for %s" slang);
+  (* Get all files then check extensions *)
+  let pattern = spf "%s/**/*" !!dir in
+  let files = Common2.glob pattern in
+  if files =*= [] then
+    failwith (spf "Empty set of parsing tests for %s at %s" slang pattern);
   List.iter check_ext files;
   let tests = parsing_tests_for_lang error_tolerance files lang in
   (match subcategory with
@@ -125,55 +130,11 @@ let pack_parsing_tests_for_lang ?(error_tolerance = Strict) lang =
 (* Note that here we also use tree-sitter to parse; certain files were not
  * parsing with pfff but parses here
  *)
-let lang_parsing_tests () =
-  Testo.categorize_suites "lang parsing"
-    [
-      (* languages with only a tree-sitter parser *)
-      pack_parsing_tests_for_lang Lang.Bash;
-      pack_parsing_tests_for_lang Lang.Csharp;
-      pack_parsing_tests_for_lang Lang.Dockerfile;
-      pack_parsing_tests_for_lang Lang.Lua;
-      pack_parsing_tests_for_lang Lang.Move_on_aptos;
-      pack_parsing_tests_for_lang Lang.Circom;
-      pack_parsing_tests_for_lang Lang.Rust;
-      pack_parsing_tests_for_lang Lang.Cairo;
-      pack_parsing_tests_for_lang Lang.Swift;
-      pack_parsing_tests_for_lang Lang.Kotlin;
-      pack_parsing_tests_for_lang Lang.Hack;
-      pack_parsing_tests_for_lang Lang.Html;
-      pack_parsing_tests_for_lang Lang.Xml;
-      pack_parsing_tests_for_lang Lang.Vue;
-      pack_parsing_tests_for_lang Lang.R;
-      pack_parsing_tests_for_lang Lang.Solidity;
-      pack_parsing_tests_for_lang Lang.Julia;
-      pack_parsing_tests_for_lang Lang.Jsonnet;
-      pack_parsing_tests_for_lang Lang.Dart;
-      (* here we have both a Pfff and tree-sitter parser *)
-      pack_parsing_tests_for_lang Lang.Java;
-      pack_parsing_tests_for_lang Lang.Go;
-      pack_parsing_tests_for_lang Lang.Ruby;
-      pack_parsing_tests_for_lang Lang.Js;
-      pack_parsing_tests_for_lang Lang.C;
-      pack_parsing_tests_for_lang Lang.Cpp;
-      pack_parsing_tests_for_lang Lang.Php;
-      pack_parsing_tests_for_lang Lang.Ocaml;
-      (* recursive descent parser *)
-      pack_parsing_tests_for_lang Lang.Scala;
-      pack_parsing_tests_for_lang Lang.Clojure;
-      pack_parsing_tests_for_lang Lang.Protobuf;
-      pack_parsing_tests_for_lang Lang.Promql;
-      pack_parsing_tests_for_lang Lang.Terraform;
-      (* a few parsing tests where we expect some partials
-       * See cpp/parsing_partial/
-       *)
-      pack_parsing_tests_for_lang ~error_tolerance:Partial_parsing Lang.Cpp;
-      (* a few parsing tests where we rely on "missing tokens" being
-         inserted by tree-sitter.
-         See cpp/parsing_missing/
-      *)
-      pack_parsing_tests_for_lang ~error_tolerance:Missing_tokens Lang.C;
-      pack_parsing_tests_for_lang ~error_tolerance:Missing_tokens Lang.Cpp;
-    ]
+let lang_parsing_tests langs_with_error_tolerance : Testo.t list =
+  langs_with_error_tolerance
+  |> List_.map (fun (lang, error_tolerance) ->
+         pack_parsing_tests_for_lang ~error_tolerance lang)
+  |> Testo.categorize_suites "lang parsing"
 
 (* It's important that our parsers generate classic parsing errors
  * exns (e.g., Parsing_error, Lexical_error), otherwise semgrep
@@ -241,11 +202,64 @@ let parsing_rules_with_atd_tests () =
 (* Tests *)
 (*****************************************************************************)
 
-let tests () =
+let make_tests langs_with_tolerance =
   List_.flatten
     [
-      lang_parsing_tests ();
+      lang_parsing_tests langs_with_tolerance;
       parsing_error_tests ();
       parsing_rules_tests ();
       parsing_rules_with_atd_tests ();
     ]
+
+let langs_with_error_tolerance =
+  [
+    (* languages with only a tree-sitter parser *)
+    (Lang.Bash, Strict);
+    (Lang.Csharp, Strict);
+    (Lang.Dockerfile, Strict);
+    (Lang.Lua, Strict);
+    (Lang.Lua, Todo);
+    (Lang.Move_on_aptos, Strict);
+    (Lang.Circom, Strict);
+    (Lang.Rust, Strict);
+    (Lang.Cairo, Strict);
+    (Lang.Swift, Strict);
+    (Lang.Kotlin, Strict);
+    (Lang.Hack, Strict);
+    (Lang.Html, Strict);
+    (Lang.Xml, Strict);
+    (Lang.Vue, Strict);
+    (Lang.R, Strict);
+    (Lang.Solidity, Strict);
+    (Lang.Julia, Strict);
+    (Lang.Jsonnet, Strict);
+    (Lang.Dart, Strict);
+    (* here we have both a Pfff and tree-sitter parser *)
+    (Lang.Java, Strict);
+    (Lang.Go, Strict);
+    (Lang.Ruby, Strict);
+    (Lang.Js, Strict);
+    (Lang.C, Strict);
+    (Lang.Cpp, Strict);
+    (Lang.Php, Strict);
+    (Lang.Ocaml, Strict);
+    (* recursive descent parser *)
+    (Lang.Scala, Strict);
+    (Lang.Clojure, Strict);
+    (Lang.Protobuf, Strict);
+    (Lang.Protobuf, Todo);
+    (Lang.Promql, Strict);
+    (Lang.Terraform, Strict);
+    (* a few parsing tests where we expect some partials
+     * See cpp/parsing_partial/
+     *)
+    (Lang.Cpp, Partial_parsing);
+    (* a few parsing tests where we rely on "missing tokens" being
+       inserted by tree-sitter.
+       See cpp/parsing_missing/
+    *)
+    (Lang.C, Missing_tokens);
+    (Lang.Cpp, Missing_tokens);
+  ]
+
+let tests () = make_tests langs_with_error_tolerance
diff --git a/src/parsing/Unit_parsing.mli b/src/parsing/Unit_parsing.mli
index 52f29b38a1fc..8c558f249b1a 100644
--- a/src/parsing/Unit_parsing.mli
+++ b/src/parsing/Unit_parsing.mli
@@ -1,2 +1,17 @@
 (* The test suite is generated by scanning the file system *)
+
 val tests : unit -> Testo.t list
+
+type error_tolerance =
+  | Strict
+  (* expect "missing tokens" inserted by the tree-sitter parser *)
+  | Missing_tokens
+  (* expect error nodes created by the tree-sitter parser *)
+  | Partial_parsing
+  (* expect some error, test status will be XFAIL *)
+  | Todo
+
+(* For semgrep-proprietary: make a test suite for a specific list of
+   languages.
+*)
+val make_tests : (Lang.t * error_tolerance) list -> Testo.t list
diff --git a/src/parsing_languages/Parse_pattern2.ml b/src/parsing_languages/Parse_pattern2.ml
index 0d07f5b317a4..77748e73f1cd 100644
--- a/src/parsing_languages/Parse_pattern2.ml
+++ b/src/parsing_languages/Parse_pattern2.ml
@@ -135,7 +135,12 @@ let parse_pattern options lang str =
       QL_to_generic.any pattern
   (* Tree-sitter only and directly to generic AST *)
   | Lang.Csharp ->
-      let res = Parse_csharp_tree_sitter.parse_pattern str in
+      let parse_pattern =
+        if Parsing_plugin.Csharp.is_available () then
+          Parsing_plugin.Csharp.parse_pattern
+        else Parse_csharp_tree_sitter.parse_pattern
+      in
+      let res = parse_pattern str in
       extract_pattern_from_tree_sitter_result res
   | Lang.Cairo ->
       let res = Parse_cairo_tree_sitter.parse_pattern str in
@@ -202,6 +207,8 @@ let parse_pattern options lang str =
   | Lang.Circom ->
       let res = Parse_circom_tree_sitter.parse_pattern str in
       extract_pattern_from_tree_sitter_result res
+  (* TODO *)
+  | Lang.Move_on_sui -> failwith "Unimplemented"
 
 let dump_tree_sitter_pattern_cst (lang : Lang.t) (path : Fpath.t) : unit =
   let file = !!path in
diff --git a/src/parsing_languages/Parse_target2.ml b/src/parsing_languages/Parse_target2.ml
index b8227c0beb6c..fd09e3024f4c 100644
--- a/src/parsing_languages/Parse_target2.ml
+++ b/src/parsing_languages/Parse_target2.ml
@@ -184,8 +184,6 @@ let just_parse_with_lang lang file : Parsing_result2.t =
    *)
   | Lang.Cairo ->
       run file [ TreeSitter Parse_cairo_tree_sitter.parse ] (fun x -> x)
-  | Lang.Csharp ->
-      run file [ TreeSitter Parse_csharp_tree_sitter.parse ] (fun x -> x)
   | Lang.Ruby ->
       run file
         [ TreeSitter Parse_ruby_tree_sitter.parse ]
@@ -230,4 +228,14 @@ let just_parse_with_lang lang file : Parsing_result2.t =
    * for parsing to take place.
    *)
   | Lang.Apex -> run_external_parser file Parsing_plugin.Apex.parse_target
+  | Lang.Csharp ->
+      let parse_target =
+        (* Use the proprietary parser if available *)
+        if Parsing_plugin.Csharp.is_available () then
+          Parsing_plugin.Csharp.parse_target
+        else Parse_csharp_tree_sitter.parse
+      in
+      run file [ TreeSitter parse_target ] (fun x -> x)
   | Lang.Elixir -> run_external_parser file Parsing_plugin.Elixir.parse_target
+  (* TODO *)
+  | Lang.Move_on_sui -> failwith "Unimplemented"
diff --git a/src/printing/Pretty_print_AST.ml b/src/printing/Pretty_print_AST.ml
index c123214e516d..33b89466627f 100644
--- a/src/printing/Pretty_print_AST.ml
+++ b/src/printing/Pretty_print_AST.ml
@@ -100,6 +100,7 @@ let _lang_kind = function
   | Lang.Rust
   | Lang.Move_on_aptos ->
       CLikeSemiColon
+  | Lang.Move_on_sui -> failwith "Unimplemented"
   | _other_ -> Other
 
 (*****************************************************************************)
@@ -167,7 +168,8 @@ let rec stmt env st =
   | DirectiveStmt _
   | DisjStmt (_, _)
   | OtherStmtWithStmt (_, _, _)
-  | OtherStmt (_, _) ->
+  | OtherStmt (_, _)
+  | RawStmt _ ->
       todo (S st)
 
 and label_ident env lbl =
@@ -256,6 +258,7 @@ and if_stmt env (tok, e, s, sopt) =
     | Lang.Swift ->
         (paren_cond, "else if", bracket_body)
     | Lang.Lua -> (paren_cond, "elseif", bracket_body)
+    | Lang.Move_on_sui -> failwith "Unimplemented"
   in
   let e_str = format_cond tok (condition env e) in
   let s_str = stmt { env with level = env.level + 1 } s in
@@ -333,6 +336,7 @@ and while_stmt env (tok, e, s) =
     | Lang.Go -> go_while
     | Lang.Ruby -> ruby_while
     | Lang.Ocaml -> ocaml_while
+    | Lang.Move_on_sui -> failwith "Unimplemented"
   in
   while_format (token ~d:"while" tok) (condition env e)
     (stmt { env with level = env.level + 1 } s)
@@ -386,6 +390,7 @@ and do_while stmt env (s, e) =
     | Lang.Rust
     | Lang.R ->
         failwith "impossible; no do while"
+    | Lang.Move_on_sui -> failwith "Unimplemented"
     | Lang.Ruby -> failwith "ruby is so weird (here, do while loop)"
   in
   do_while_format (stmt { env with level = env.level + 1 } s) (expr env e)
@@ -440,6 +445,7 @@ and for_stmt env (for_tok, hdr, s) =
     | Lang.Ocaml
     | Lang.Ql ->
         failwith "JSON/OCaml/QL has for loops????"
+    | Lang.Move_on_sui -> failwith "Unimplemented"
   in
   let show_init = function
     | ForInitVar (ent, var_def) ->
@@ -551,6 +557,7 @@ and def_stmt env (entity, def_kind) =
       | Lang.Jsonnet
       | Lang.Ocaml ->
           failwith "I think JSON/OCaml have no variable definitions"
+      | Lang.Move_on_sui -> failwith "Unimplemented"
     in
     let typ, id =
       match ent.name with
diff --git a/src/rpc/RPC.ml b/src/rpc/RPC.ml
index 021d5a8e5910..b94bafcf21cc 100644
--- a/src/rpc/RPC.ml
+++ b/src/rpc/RPC.ml
@@ -75,6 +75,16 @@ let handle_call (caps : < Cap.exec ; Cap.tmp >) :
           Error
             "Dependency resolution is a proprietary feature, but semgrep-pro \
              has not been loaded")
+  | `CallDumpRulePartitions params -> (
+      match !RPC_return.hook_dump_rule_partitions with
+      | Some dump_rule_partitions ->
+          let Out.{ rules; n_partitions; output_dir } = params in
+          let ok = dump_rule_partitions rules n_partitions output_dir in
+          Ok (`RetDumpRulePartitions ok)
+      | None ->
+          Error
+            "Dump rule partitions is a proprietary feature, but semgreep-pro \
+             has not been loaded")
 
 (*****************************************************************************)
 (* Helpers *)
diff --git a/src/rpc/RPC_return.ml b/src/rpc/RPC_return.ml
index a742031c1d40..b21748d6edf8 100644
--- a/src/rpc/RPC_return.ml
+++ b/src/rpc/RPC_return.ml
@@ -101,3 +101,4 @@ let validate (path : Out.fpath) : bool =
 (* Hooks for handlers defined in Pro_RPC_return.ml *)
 (*****************************************************************************)
 let hook_resolve_dependencies = ref None
+let hook_dump_rule_partitions = ref None
diff --git a/src/rpc/RPC_return.mli b/src/rpc/RPC_return.mli
index 7fb05fe9e704..e0bf33a3629b 100644
--- a/src/rpc/RPC_return.mli
+++ b/src/rpc/RPC_return.mli
@@ -22,3 +22,6 @@ val hook_resolve_dependencies :
   (Out.manifest * Out.resolution_result) list)
   option
   ref
+
+val hook_dump_rule_partitions :
+  (Out.raw_json -> int -> Fpath.t -> bool) option ref
diff --git a/src/rule/Lang.ml b/src/rule/Lang.ml
index ab9b9161e15a..c40376bde26f 100644
--- a/src/rule/Lang.ml
+++ b/src/rule/Lang.ml
@@ -63,6 +63,7 @@ type t = Language.t =
   | Kotlin
   | Lisp
   | Lua
+  | Move_on_sui
   | Move_on_aptos
   | Ocaml
   | Php
diff --git a/src/rule/Lang.mli b/src/rule/Lang.mli
index e774ff5ce531..5f4d92f45136 100644
--- a/src/rule/Lang.mli
+++ b/src/rule/Lang.mli
@@ -37,6 +37,7 @@ type t = Language.t =
   | Kotlin
   | Lisp
   | Lua
+  | Move_on_sui
   | Move_on_aptos
   | Ocaml
   | Php
diff --git a/src/rule/Rule.ml b/src/rule/Rule.ml
index a0f610b0cd6c..b2c3e67c51fa 100644
--- a/src/rule/Rule.ml
+++ b/src/rule/Rule.ml
@@ -142,18 +142,18 @@ and metavar_cond =
 and metavar_cond_name = {
   mvar : Mvar.t;
   kind : metavar_name_kind option;
-  module_ : string option;
-      (** A user-facing module name.
+  modules : string list option;
+      (** A list of user-facing module names.
 
         Internally we would further resolve this in the relevant hook. Mainly
         for JS/TS, where what is used for require or import might have
         non-identifier characters, and there aren't really fully qualified
-        names. For instance, this might be `foo.bar`, or `@lib/something`, or
-        `@angular`.
+        names. For instance, elements of this might be `foo.bar`, or
+        `@lib/something`, or `@angular`.
 
-        Applicable for other languages too, like for Java, e.g., this could be
-        `com.foo.bar`, but less important since those patterns may be able to
-        be written directly.
+        Applicable for other languages too, like for Java, e.g., elements could
+        be `com.foo.bar`, but less important since those patterns may be able
+        to be written directly.
   *)
 }
 
diff --git a/src/tainting/Dataflow_tainting.ml b/src/tainting/Dataflow_tainting.ml
index 614e5b152fad..00202b574763 100644
--- a/src/tainting/Dataflow_tainting.ml
+++ b/src/tainting/Dataflow_tainting.ml
@@ -26,8 +26,8 @@ module LV = IL_helpers
 module T = Taint
 module Lval_env = Taint_lval_env
 module Taints = T.Taint_set
-module TM = Taint_smatch
-open Shape_and_sig.Shape
+module TM = Taint_spec_match
+module S = Shape_and_sig.Shape
 module Shape = Taint_shape
 module Effect = Shape_and_sig.Effect
 module Signature = Shape_and_sig.Signature
@@ -80,6 +80,8 @@ type a_propagator = {
   var : var;
 }
 
+type effects_handler = var option -> Shape_and_sig.Effect.t list -> unit
+
 type config = {
   filepath : string;
   rule_id : Rule_ID.t;
@@ -94,7 +96,7 @@ type config = {
        * `sanitize(sink(tainted))` will not yield any finding.
        * *)
   unify_mvars : bool;
-  handle_effects : var option -> Effect.t list -> Lval_env.t -> unit;
+  handle_effects : effects_handler;
 }
 
 type mapping = Lval_env.t D.mapping
@@ -262,8 +264,8 @@ let taints_of_matches env ~incoming sources =
   (data_taints, lval_env)
 
 let report_effects env effects =
-  if effects <> [] then
-    env.config.handle_effects env.fun_name effects env.lval_env
+  if not (List_.null effects) then
+    env.config.handle_effects env.fun_name effects
 
 let unify_mvars_sets env mvars1 mvars2 =
   let xs =
@@ -365,8 +367,7 @@ let partition_sources_by_side_effect sources_matches =
  * (there is just no point in doing so). *)
 let get_control_taints_to_return env =
   Lval_env.get_control_taints env.lval_env
-  |> Taints.elements
-  |> List.filter (fun ({ orig; _ } : T.taint) ->
+  |> Taints.filter (fun ({ orig; _ } : T.taint) ->
          match orig with
          | T.Src _ -> true
          | Var _
@@ -449,7 +450,7 @@ let propagate_taint_to_label replace_labels label (taint : T.taint) =
   { taint with orig = new_orig }
 
 (*****************************************************************************)
-(* Reporting effects *)
+(* Effects and signatures *)
 (*****************************************************************************)
 
 (* Potentially produces an effect from incoming taints + call traces to a sink.
@@ -567,11 +568,10 @@ let effects_of_tainted_return env taints shape return_tok : Effect.t list =
   let control_taints = get_control_taints_to_return env in
   if
     Shape.taints_and_shape_are_relevant taints shape
-    || not (List_.null control_taints)
+    || not (Taints.is_empty control_taints)
   then
     let data_taints =
-      taints |> Taints.elements
-      |> List_.map (fun t -> { t with T.tokens = List.rev t.T.tokens })
+      taints |> Taints.map (fun t -> { t with T.tokens = List.rev t.T.tokens })
     in
     [
       Effect.ToReturn
@@ -579,6 +579,15 @@ let effects_of_tainted_return env taints shape return_tok : Effect.t list =
     ]
   else []
 
+let lookup_signature env fun_exp =
+  match (!hook_function_taint_signature, fun_exp) with
+  | Some hook, { e = Fetch _f; eorig = SameAs eorig } -> hook env.config eorig
+  | __else__ -> None
+
+(*****************************************************************************)
+(* Miscellaneous *)
+(*****************************************************************************)
+
 let check_orig_if_sink env ?filter_sinks orig taints shape =
   (* NOTE(gather-all-taints):
    * A sink is something opaque to us, e.g. consider sink(["ok", "tainted"]),
@@ -598,10 +607,6 @@ let check_orig_if_sink env ?filter_sinks orig taints shape =
   let effects = effects_of_tainted_sinks env taints sinks in
   report_effects env effects
 
-(*****************************************************************************)
-(* Miscellaneous large functions *)
-(*****************************************************************************)
-
 let fix_poly_taint_with_field env lval xtaint =
   let type_of_il_offset il_offset =
     match il_offset.IL.o with
@@ -878,7 +883,7 @@ let find_lval_taint_sources env incoming_taints lval =
   (taints_to_return, lval_env)
 
 let rec check_tainted_lval env (lval : IL.lval) :
-    Taints.t * shape * [ `Sub of Taints.t * shape ] * Lval_env.t =
+    Taints.t * S.shape * [ `Sub of Taints.t * S.shape ] * Lval_env.t =
   let new_taints, lval_in_env, lval_shape, sub, lval_env =
     check_tainted_lval_aux env lval
   in
@@ -976,8 +981,8 @@ and propagate_taint_via_java_getters_and_setters_without_definition env e args
 and check_tainted_lval_aux env (lval : IL.lval) :
     Taints.t
     * Xtaint.t_or_sanitized
-    * shape
-    * [ `Sub of Taints.t * shape ]
+    * S.shape
+    * [ `Sub of Taints.t * S.shape ]
     * Lval_env.t =
   (* Recursively checks an l-value bottom-up.
    *
@@ -1040,13 +1045,13 @@ and check_tainted_lval_aux env (lval : IL.lval) :
         match sub_in_env with
         | `Sanitized ->
             (* See NOTE [lval/sanitized] *)
-            (`Sanitized, Bot)
+            (`Sanitized, S.Bot)
         | (`Clean | `None | `Tainted _) as sub_xtaint ->
             let xtaint', shape =
               (* THINK: Should we just use 'Sig.find_in_shape' directly here ?
                        We have the 'sub_shape' available. *)
               match Lval_env.find_lval lval_env lval with
-              | None -> (`None, Bot)
+              | None -> (`None, S.Bot)
               | Some (Cell (xtaint', shape)) -> (xtaint', shape)
             in
             let xtaint' =
@@ -1147,7 +1152,7 @@ and check_tainted_lval_offset env offset =
 
 (* Test whether an expression is tainted, and if it is also a sink,
  * report the finding too (by side effect). *)
-and check_tainted_expr env exp : Taints.t * shape * Lval_env.t =
+and check_tainted_expr env exp : Taints.t * S.shape * Lval_env.t =
   let check env = check_tainted_expr env in
   let check_subexpr exp =
     match exp.e with
@@ -1155,20 +1160,20 @@ and check_tainted_expr env exp : Taints.t * shape * Lval_env.t =
     (* TODO: 'Fetch' is handled specially, this case should not never be taken.  *)
     | Literal _
     | FixmeExp (_, _, None) ->
-        (Taints.empty, Bot, env.lval_env)
+        (Taints.empty, S.Bot, env.lval_env)
     | FixmeExp (_, _, Some e) ->
         let taints, shape, lval_env = check env e in
         let taints =
           taints |> Taints.union (Shape.gather_all_taints_in_shape shape)
         in
-        (taints, Bot, lval_env)
+        (taints, S.Bot, lval_env)
     | Composite ((CTuple | CArray | CList), (_, es, _)) ->
         let taints_and_shapes, lval_env = map_check_expr env check es in
         let obj = Shape.tuple_like_obj taints_and_shapes in
         (Taints.empty, Obj obj, lval_env)
     | Composite ((CSet | Constructor _ | Regexp), (_, es, _)) ->
         let taints, lval_env = union_map_taints_and_vars env check es in
-        (taints, Bot, lval_env)
+        (taints, S.Bot, lval_env)
     | Operator ((op, _), es) ->
         let args_taints, all_args_taints, lval_env =
           check_function_call_arguments env es
@@ -1237,7 +1242,7 @@ and check_tainted_expr env exp : Taints.t * shape * Lval_env.t =
           | G.RSA ->
               all_args_taints
         in
-        (op_taints, Bot, lval_env)
+        (op_taints, S.Bot, lval_env)
     | RecordOrDict fields ->
         (* TODO: Construct a proper record/dict shape here. *)
         let fields_exprs =
@@ -1251,7 +1256,7 @@ and check_tainted_expr env exp : Taints.t * shape * Lval_env.t =
         let taints, lval_env =
           union_map_taints_and_vars env check fields_exprs
         in
-        (taints, Bot, lval_env)
+        (taints, S.Bot, lval_env)
     | Cast (_, e) -> check env e
   in
   match exp_is_sanitized env exp with
@@ -1315,7 +1320,7 @@ and check_function_call_arguments env args =
   let all_args_taints = List.fold_left Taints.union Taints.empty rev_taints in
   (args_taints, all_args_taints, lval_env)
 
-let check_tainted_var env (var : IL.name) : Taints.t * shape * Lval_env.t =
+let check_tainted_var env (var : IL.name) : Taints.t * S.shape * Lval_env.t =
   let taints, shape, _sub, lval_env =
     check_tainted_lval env (LV.lval_of_var var)
   in
@@ -1329,11 +1334,10 @@ let check_tainted_var env (var : IL.name) : Taints.t * shape * Lval_env.t =
       input into the function body, from the calling context?
 *)
 let check_function_call env fun_exp args
-    (args_taints : (Taints.t * shape) argument list) :
-    (Taints.t * shape * Lval_env.t) option =
-  match (!hook_function_taint_signature, fun_exp) with
-  | Some hook, { e = Fetch _f; eorig = SameAs eorig } ->
-      let* fparams, fun_sig = hook env.config eorig in
+    (args_taints : (Taints.t * S.shape) argument list) :
+    (Taints.t * S.shape * Lval_env.t) option =
+  match lookup_signature env fun_exp with
+  | Some fun_sig ->
       Log.debug (fun m ->
           m ~tags:sigs_tag "Call to %s : %s"
             (Display_IL.string_of_exp fun_exp)
@@ -1343,29 +1347,40 @@ let check_function_call env fun_exp args
         (taints, shape)
       in
       let* call_effects =
-        Sig_inst.instantiate_function_signature env.lval_env ~check_lval fparams
-          fun_sig fun_exp eorig args args_taints
+        Sig_inst.instantiate_function_signature env.lval_env ~check_lval fun_sig
+          ~callee:fun_exp ~args:(Some args) args_taints
       in
       Some
         (call_effects
         |> List.fold_left
-             (fun (taints_acc, shape_acc, lval_env) effect ->
-               match effect with
-               | `ToSink (incoming_taints, sink) ->
+             (fun (taints_acc, shape_acc, lval_env)
+                  (call_effect : Sig_inst.call_effect) ->
+               match call_effect with
+               | ToSink
+                   {
+                     taints_with_precondition = incoming_taints, _requires;
+                     sink;
+                     _;
+                   } ->
                    effects_of_tainted_sink env incoming_taints sink
                    |> report_effects env;
                    (taints_acc, shape_acc, lval_env)
-               | `ToReturn (taints, shape, control_taints, _return_tok) ->
+               | ToReturn
+                   {
+                     data_taints = taints;
+                     data_shape = shape;
+                     control_taints;
+                     return_tok = _;
+                   } ->
                    ( Taints.union taints taints_acc,
                      Shape.unify_shape shape shape_acc,
                      Lval_env.add_control_taints lval_env control_taints )
-               | `ToLval (taints, lval) ->
+               | ToLval (taints, lval) ->
                    (taints_acc, shape_acc, lval_env |> Lval_env.add lval taints))
              (Taints.empty, Bot, env.lval_env))
-  | None, _
-  | Some _, _ ->
+  | None ->
       Log.debug (fun m ->
-          m ~tags:sigs_tag "Call to %s : NO SIGNATURE"
+          m ~tags:sigs_tag "Call to %s : NO SIGNATURE !!!!"
             (Display_IL.string_of_exp fun_exp));
       None
 
@@ -1387,7 +1402,7 @@ let check_function_call_callee env e =
 
 (* Test whether an instruction is tainted, and if it is also a sink,
  * report the effect too (by side effect). *)
-let check_tainted_instr env instr : Taints.t * shape * Lval_env.t =
+let check_tainted_instr env instr : Taints.t * S.shape * Lval_env.t =
   let check_expr env = check_tainted_expr env in
   let check_instr = function
     | Assign (_, e) ->
@@ -1405,7 +1420,7 @@ let check_tainted_instr env instr : Taints.t * shape * Lval_env.t =
           all_args_taints
           |> Taints.union (gather_all_taints_in_args_taints args_taints)
         in
-        let e_obj, e_taints, _e_shape, lval_env =
+        let e_obj, e_taints, _e_shape_TODO, lval_env =
           check_function_call_callee { env with lval_env } e
         in
         (* NOTE(sink_has_focus):
@@ -1426,11 +1441,10 @@ let check_tainted_instr env instr : Taints.t * shape * Lval_env.t =
           | Some (call_taints, shape, lval_env) ->
               (* THINK: For debugging, we could print a diff of the previous and new lval_env *)
               Log.debug (fun m ->
-                  m ~tags:sigs_tag
-                    "Instantiating taint signature of %s: returns %s & %s"
+                  m ~tags:sigs_tag "- Instantiating %s: returns %s & %s"
                     (Display_IL.string_of_exp e)
                     (T.show_taints call_taints)
-                    (show_shape shape));
+                    (S.show_shape shape));
               (call_taints, shape, lval_env)
           | None -> (
               let call_taints =
@@ -1558,7 +1572,7 @@ let check_tainted_instr env instr : Taints.t * shape * Lval_env.t =
 
 (* Test whether a `return' is tainted, and if it is also a sink,
  * report the effect too (by side effect). *)
-let check_tainted_return env tok e : Taints.t * shape * Lval_env.t =
+let check_tainted_return env tok e : Taints.t * S.shape * Lval_env.t =
   let sinks =
     any_is_best_sink env (G.Tk tok) @ orig_is_best_sink env e.eorig
     |> List.filter (TM.is_best_match env.best_matches)
@@ -1604,8 +1618,7 @@ let effects_from_arg_updates_at_exit enter_env exit_env : Effect.t list =
                         let new_taints = Taints.diff exit_taints enter_taints in
                         (* TODO: Also report if taints are _cleaned_. *)
                         if not (Taints.is_empty new_taints) then
-                          Some
-                            (Effect.ToLval (new_taints |> Taints.elements, lval))
+                          Some (Effect.ToLval (new_taints, lval))
                         else None)))
   |> Seq.concat |> List.of_seq
 
@@ -1663,33 +1676,16 @@ let input_env ~enter_env ~(flow : F.cfg) mapping ni =
       | [ penv ] -> penv
       | penv1 :: penvs -> List.fold_left Lval_env.union penv1 penvs)
 
-let transfer :
-    Lang.t ->
-    Rule_options.t ->
-    config ->
-    Lval_env.t ->
-    string option ->
-    flow:F.cfg ->
-    best_matches:TM.Best_matches.t ->
-    java_props:java_props_cache ->
-    Lval_env.t D.transfn =
- fun lang options config enter_env opt_name ~flow ~best_matches ~java_props
+let transfer : env -> flow:F.cfg -> Lval_env.t D.transfn =
+ fun enter_env ~flow
      (* the transfer function to update the mapping at node index ni *)
        mapping ni ->
   (* DataflowX.display_mapping flow mapping show_tainted; *)
-  let in' : Lval_env.t = input_env ~enter_env ~flow mapping ni in
-  let node = flow.graph#nodes#assoc ni in
-  let env =
-    {
-      lang;
-      options;
-      config;
-      fun_name = opt_name;
-      lval_env = in';
-      best_matches;
-      java_props;
-    }
+  let in' : Lval_env.t =
+    input_env ~enter_env:enter_env.lval_env ~flow mapping ni
   in
+  let node = flow.graph#nodes#assoc ni in
+  let env = { enter_env with lval_env = in' } in
   let out' : Lval_env.t =
     match node.F.n with
     | NInstr x ->
@@ -1799,7 +1795,7 @@ let (fixpoint :
       mapping) =
  fun ?in_env ?name:opt_name lang options config java_props flow ->
   let init_mapping = DataflowX.new_node_array flow Lval_env.empty_inout in
-  let enter_env =
+  let enter_lval_env =
     match in_env with
     | None -> Lval_env.empty
     | Some in_env -> in_env
@@ -1831,18 +1827,25 @@ let (fixpoint :
         sources |> Seq.append sanitizers |> Seq.append sinks)
       flow
   in
+  let env =
+    {
+      lang;
+      options;
+      config;
+      fun_name = opt_name;
+      lval_env = enter_lval_env;
+      best_matches;
+      java_props;
+    }
+  in
   (* THINK: Why I cannot just update mapping here ? if I do, the mapping gets overwritten later on! *)
   (* DataflowX.display_mapping flow init_mapping show_tainted; *)
   let end_mapping =
     DataflowX.fixpoint ~timeout:Limits_semgrep.taint_FIXPOINT_TIMEOUT
-      ~eq_env:Lval_env.equal ~init:init_mapping
-      ~trans:
-        (transfer lang options config enter_env opt_name ~flow ~best_matches
-           ~java_props)
-        (* tainting is a forward analysis! *)
+      ~eq_env:Lval_env.equal ~init:init_mapping ~trans:(transfer env ~flow)
       ~forward:true ~flow
   in
-  let exit_env = end_mapping.(flow.exit).D.out_env in
-  ( effects_from_arg_updates_at_exit enter_env exit_env |> fun effects ->
-    if effects <> [] then config.handle_effects opt_name effects exit_env );
+  let exit_lval_env = end_mapping.(flow.exit).D.out_env in
+  effects_from_arg_updates_at_exit enter_lval_env exit_lval_env
+  |> report_effects env;
   end_mapping
diff --git a/src/tainting/Dataflow_tainting.mli b/src/tainting/Dataflow_tainting.mli
index 260937d2552b..261fd4fd559e 100644
--- a/src/tainting/Dataflow_tainting.mli
+++ b/src/tainting/Dataflow_tainting.mli
@@ -7,15 +7,20 @@ type a_propagator = {
   var : var; (* REMOVE USE prop.id *)
 }
 
+type effects_handler =
+  var option (** function name ('None' if anonymous) *) ->
+  Shape_and_sig.Effect.t list ->
+  unit
+
 type config = {
   filepath : string;  (** File under analysis, for Deep Semgrep. *)
   rule_id : Rule_ID.t;  (** Taint rule id, for Deep Semgrep. *)
   track_control : bool;
       (** Whether the rule requires tracking "control taint". *)
-  is_source : AST_generic.any -> Rule.taint_source Taint_smatch.t list;
+  is_source : AST_generic.any -> Rule.taint_source Taint_spec_match.t list;
       (** Test whether 'any' is a taint source, this corresponds to
       * 'pattern-sources:' in taint-mode. *)
-  is_propagator : AST_generic.any -> a_propagator Taint_smatch.t list;
+  is_propagator : AST_generic.any -> a_propagator Taint_spec_match.t list;
       (** Test whether 'any' matches a taint propagator, this corresponds to
        * 'pattern-propagators:' in taint-mode.
        *
@@ -45,19 +50,15 @@ type config = {
        * anyhow it's clearly incorrect to taint `Shell`, so a better solution was
        * needed (hence `pattern-propagators`).
        *)
-  is_sink : AST_generic.any -> Rule.taint_sink Taint_smatch.t list;
+  is_sink : AST_generic.any -> Rule.taint_sink Taint_spec_match.t list;
       (** Test whether 'any' is a sink, this corresponds to 'pattern-sinks:'
       * in taint-mode. *)
-  is_sanitizer : AST_generic.any -> Rule.taint_sanitizer Taint_smatch.t list;
+  is_sanitizer :
+    AST_generic.any -> Rule.taint_sanitizer Taint_spec_match.t list;
       (** Test whether 'any' is a sanitizer, this corresponds to
       * 'pattern-sanitizers:' in taint-mode. *)
   unify_mvars : bool;  (** Unify metavariables in sources and sinks? *)
-  handle_effects :
-    var option (** function name ('None' if anonymous) *) ->
-    Shape_and_sig.Effect.t list ->
-    Taint_lval_env.t ->
-    unit;
-      (** Callback to report effects. *)
+  handle_effects : effects_handler;  (** Callback to report effects. *)
 }
 (** Taint rule instantiated for a given file.
   *
@@ -80,12 +81,7 @@ type java_props_cache
 val mk_empty_java_props_cache : unit -> java_props_cache
 
 val hook_function_taint_signature :
-  (config ->
-  AST_generic.expr ->
-  (AST_generic.parameters (* params of function *) * Shape_and_sig.Signature.t)
-  option)
-  option
-  ref
+  (config -> AST_generic.expr -> Shape_and_sig.Signature.t option) option ref
 (** Pro inter-file (aka deep) *)
 
 val hook_find_attribute_in_class :
diff --git a/src/tainting/Shape_and_sig.ml b/src/tainting/Shape_and_sig.ml
index cfe0caf19a82..6c7a5c6fe4d7 100644
--- a/src/tainting/Shape_and_sig.ml
+++ b/src/tainting/Shape_and_sig.ml
@@ -33,7 +33,7 @@ end)
  * associated with its fields and indexes.
  *
  * Taint shapes are a bit like types. Right now this is mainly to support
- * field- and index-sensitivity, but hapes also provide a good foundation to
+ * field- and index-sensitivity, but shapes also provide a good foundation to
  * later add alias analysis.  This is somewhat inspired by
  *
  *     "Polymorphic type, region and effect inference"
@@ -249,10 +249,10 @@ and Effect : sig
   }
 
   type taints_to_return = {
-    data_taints : Taint.taint list;
+    data_taints : Taint.taints;
         (** The taints of the data being returned (typical data propagated via data flow). *)
     data_shape : Shape.shape;  (** The shape of the data being returned. *)
-    control_taints : Taint.taint list;
+    control_taints : Taint.taints;
         (** The taints propagated via the control flow (cf., `control: true` sources)
    * used for reachability queries. *)
     return_tok : AST_generic.tok;
@@ -299,7 +299,7 @@ and Effect : sig
         *
         *     ToReturn(["taint"], Bot, ...)
         *)
-    | ToLval of Taint.taint list * Taint.lval
+    | ToLval of Taint.taints * Taint.lval
         (** Taints reach an l-value in the scope of the function/method.
         *
         * For example:
@@ -319,7 +319,13 @@ and Effect : sig
 
   val compare : t -> t -> int
   val show : t -> string
+
+  (* Mainly for debugging *)
+  val show_taints_to_sink : taints_to_sink -> string
+  val show_taints_to_return : taints_to_return -> string
 end = struct
+  module Taints = Taint.Taint_set
+
   type sink = { pm : Pattern_match.t; rule_sink : R.taint_sink }
   type taint_to_sink_item = { taint : T.taint; sink_trace : unit T.call_trace }
 
@@ -336,16 +342,16 @@ end = struct
   }
 
   type taints_to_return = {
-    data_taints : Taint.taint list;
+    data_taints : Taint.taints;
     data_shape : Shape.shape;
-    control_taints : Taint.taint list;
+    control_taints : Taint.taints;
     return_tok : AST_generic.tok;
   }
 
   type t =
     | ToSink of taints_to_sink
     | ToReturn of taints_to_return
-    | ToLval of T.taint list * T.lval (* TODO: CleanArg ? *)
+    | ToLval of T.taints * T.lval (* TODO: CleanArg ? *)
 
   (*************************************)
   (* Comparison *)
@@ -395,10 +401,10 @@ end = struct
         control_taints = control_taints2;
         return_tok = _;
       } =
-    match List.compare T.compare_taint data_taints1 data_taints2 with
+    match Taints.compare data_taints1 data_taints2 with
     | 0 -> (
         match Shape.compare_shape data_shape1 data_shape2 with
-        | 0 -> List.compare T.compare_taint control_taints1 control_taints2
+        | 0 -> Taints.compare control_taints1 control_taints2
         | other -> other)
     | other -> other
 
@@ -407,7 +413,7 @@ end = struct
     | ToSink tts1, ToSink tts2 -> compare_taints_to_sink tts1 tts2
     | ToReturn ttr1, ToReturn ttr2 -> compare_taints_to_return ttr1 ttr2
     | ToLval (ts1, lv1), ToLval (ts2, lv2) -> (
-        match List.compare T.compare_taint ts1 ts2 with
+        match Taints.compare ts1 ts2 with
         | 0 -> T.compare_lval lv1 lv2
         | other -> other)
     | ToSink _, (ToReturn _ | ToLval _) -> -1
@@ -445,17 +451,33 @@ end = struct
   let show_taints_to_sink { taints_with_precondition = taints, _; sink; _ } =
     Common.spf "%s ~~~> %s" (show_taints_and_traces taints) (show_sink sink)
 
+  let show_taints_to_return
+      { data_taints; data_shape; control_taints; return_tok = _ } =
+    Printf.sprintf "return (%s & %s & CTRL:%s)"
+      (T.show_taints data_taints)
+      (Shape.show_shape data_shape)
+      (T.show_taints control_taints)
+
   let show = function
-    | ToSink x -> show_taints_to_sink x
-    | ToReturn { data_taints; data_shape; control_taints; return_tok = _ } ->
-        Printf.sprintf "return (%s & %s & CTRL:%s)"
-          (Common2.string_of_list T.show_taint data_taints)
-          (Shape.show_shape data_shape)
-          (Common2.string_of_list T.show_taint control_taints)
+    | ToSink tts -> show_taints_to_sink tts
+    | ToReturn ttr -> show_taints_to_return ttr
     | ToLval (taints, lval) ->
-        Printf.sprintf "%s ----> %s"
-          (Common2.string_of_list T.show_taint taints)
-          (T.show_lval lval)
+        Printf.sprintf "%s ----> %s" (T.show_taints taints) (T.show_lval lval)
+end
+
+and Effects : sig
+  include Set.S with type elt = Effect.t
+
+  val show : t -> string
+end = struct
+  include Set.Make (struct
+    type t = Effect.t
+
+    let compare effect1 effect2 = Effect.compare effect1 effect2
+  end)
+
+  let show s =
+    s |> to_seq |> List.of_seq |> List_.map Effect.show |> String.concat "; "
 end
 
 (** A (polymorphic) taint signature: simply a set of results for a function.
@@ -471,7 +493,7 @@ end
  *
  * We infer the signature (simplified):
  *
- *     {ToSink {taints_with_precondition = [(x#0).a]; sink = ... ; ...}}
+ *     x => {ToSink {taints_with_precondition = [(x#0).a]; sink = ... ; ...}}
  *
  * where '(x#0).a' is taint variable that denotes the taint of the offset `.a`
  * of the parameter `x` (where '#0' means it is the first argument) of `foo`.
@@ -488,18 +510,81 @@ end
  * THINK: Could we have a "taint shape" for functions/methods ?
  *)
 and Signature : sig
-  include Set.S with type elt = Effect.t
+  (** A simplified version of 'AST_generic.parameter', we use 'Other' to represent
+    * parameter kinds that we do not support yet. We don't want to just remove
+    * those unsupported parameters because we rely on the position of a parameter
+    * to represent taint variables, see 'Taint.arg'. *)
+  type param = P of string | Other [@@deriving eq, ord]
+
+  type params = param list [@@deriving eq, ord]
 
+  type t = { params : params; effects : Effects.t } [@@deriving eq, ord]
+  (**
+   * The 'params' act like an universal quantifier, we need them to later
+   * instantiate the accompanying signature. *)
+
+  val of_generic_params : AST_generic.parameters -> params
+  val show_params : params -> string
   val show : t -> string
 end = struct
-  include Set.Make (struct
-    type t = Effect.t
+  (*************************************)
+  (* Param(eter)s *)
+  (*************************************)
 
-    let compare r1 r2 = Effect.compare r1 r2
-  end)
+  (* TODO: Now with HOFs we run the risk of shadowing... *)
+  type param = P of string | Other
+  type params = param list
 
-  let show s =
-    s |> to_seq |> List.of_seq |> List_.map Effect.show |> String.concat " + "
+  let equal_param param1 param2 =
+    match (param1, param2) with
+    | P s1, P s2 -> String.equal s1 s2
+    | Other, Other -> true
+    | P _, Other
+    | Other, P _ ->
+        false
+
+  let compare_param param1 param2 =
+    match (param1, param2) with
+    | P s1, P s2 -> String.compare s1 s2
+    | Other, Other -> 0
+    | P _, Other -> -1
+    | Other, P _ -> 1
+
+  let show_param = function
+    | P s -> s
+    | Other -> "_?"
+
+  let equal_params params1 params2 = List.equal equal_param params1 params2
+
+  let compare_params params1 params2 =
+    List.compare compare_param params1 params2
+
+  let show_params params = params |> List_.map show_param |> String.concat ", "
+
+  let of_generic_params (_, g_params, _) =
+    g_params
+    |> List_.map (function
+         | AST_generic.Param { pname = Some (s, _); _ } -> P s
+         | __else__ -> Other)
+
+  (*************************************)
+  (* Signatures *)
+  (*************************************)
+
+  type t = { params : params; effects : Effects.t }
+
+  let equal { params = params1; effects = effects1 }
+      { params = params2; effects = effects2 } =
+    equal_params params1 params2 && Effects.equal effects1 effects2
+
+  let compare { params = params1; effects = effects1 }
+      { params = params2; effects = effects2 } =
+    match compare_params params1 params2 with
+    | 0 -> Effects.compare effects1 effects2
+    | other -> other
+
+  let show { params; effects } =
+    spf "%s => {%s}" (show_params params) (Effects.show effects)
 end
 
 module Effects_tbl = Hashtbl.Make (struct
diff --git a/src/tainting/Sig_inst.ml b/src/tainting/Sig_inst.ml
index 02d7da160968..ad0ecd31ada7 100644
--- a/src/tainting/Sig_inst.ml
+++ b/src/tainting/Sig_inst.ml
@@ -22,17 +22,72 @@ open Shape_and_sig.Shape
 module Fields = Shape_and_sig.Fields
 module Shape = Taint_shape
 module Effect = Shape_and_sig.Effect
+module Effects = Shape_and_sig.Effects
 module Signature = Shape_and_sig.Signature
 module Lval_env = Taint_lval_env
 
 let sigs_tag = Log_tainting.sigs_tag
 let bad_tag = Log_tainting.bad_tag
 
+(*****************************************************************************)
+(* Call effets *)
+(*****************************************************************************)
+
+type call_effect =
+  | ToSink of Effect.taints_to_sink
+  | ToReturn of Effect.taints_to_return
+  | ToLval of Taint.taints * IL.lval
+
+type call_effects = call_effect list
+
+let show_call_effect = function
+  | ToSink tts -> Effect.show_taints_to_sink tts
+  | ToReturn ttr -> Effect.show_taints_to_return ttr
+  | ToLval (taints, lval) ->
+      Printf.sprintf "%s ----> %s" (T.show_taints taints)
+        (Display_IL.string_of_lval lval)
+
+let show_call_effects call_effects =
+  call_effects |> List_.map show_call_effect |> String.concat "; "
+
+(*****************************************************************************)
+(* Instantiation "config" *)
+(*****************************************************************************)
+
+type inst_var = {
+  inst_lval : T.lval -> (Taints.t * shape) option;
+      (** How to instantiate a 'Taint.lval', aka "data taint variable". *)
+  inst_ctrl : unit -> Taints.t;
+      (** How to instantiate a 'Taint.Control', aka "control taint variable". *)
+}
+
+(* TODO: Right now this is only for source traces, not for sink traces...
+ * In fact, we should probably not have two traces but just one, but more
+ * general. *)
+type inst_trace = {
+  add_call_to_trace_for_src :
+    Tok.t list ->
+    Rule.taint_source T.call_trace ->
+    Rule.taint_source T.call_trace option;
+      (** For sources we extend the call trace. *)
+  fix_token_trace_for_var : var_tokens:Tok.t list -> Tok.t list -> Tok.t list;
+      (** For variables we should too, but due to limitations in our call-trace
+   * representation, we just record the path as tainted tokens. *)
+}
+
+(*****************************************************************************)
+(* Helpers *)
+(*****************************************************************************)
+
 let ( let+ ) x f =
   match x with
   | None -> []
   | Some x -> f x
 
+(*****************************************************************************)
+(* Instantiating traces *)
+(*****************************************************************************)
+
 (* Try to get an idnetifier from a callee/function expression, to be used in
  * a taint trace. *)
 let get_ident_of_callee callee =
@@ -47,75 +102,158 @@ let get_ident_of_callee callee =
       | __else__ -> None)
   | __else__ -> None
 
-(* TODO: Move to 'Taint' module ? *)
-let subst_in_precondition ~inst_lval ~inst_ctrl taint =
+let add_call_to_trace_if_callee_has_eorig ~callee tainted_tokens call_trace =
+  (* E.g. (ToReturn) the call to 'bar' in:
+   *
+   *     1 def bar():
+   *     2     x = taint
+   *     3     return x
+   *     4
+   *     5 def foo():
+   *     6     y = bar()
+   *     7     sink(y)
+   *
+   * would result in this call trace for the source:
+   *
+   *     Call('bar' @l.6, ["x" @l.2], "taint" @l.2)
+   *
+   * E.g. (ToLval) the call to 'bar' in:
+   *
+   *     1 s = set([])
+   *     2
+   *     3 def bar():
+   *     4    global s
+   *     5    s.add(taint)
+   *     6
+   *     7 def foo():
+   *     8    global s
+   *     9    bar()
+   *    10    sink(s)
+   *
+   * would result in this call trace for the source:
+   *
+   *     Call('bar' @l.6, ["s" @l.5], "taint" @l.5)
+   *)
+  match callee with
+  | { IL.e = _; eorig = SameAs orig_callee } ->
+      Some (T.Call (orig_callee, tainted_tokens, call_trace))
+  | __else__ ->
+      (* TODO: Have a better fallback in case we can't get an eorig from 'callee',
+       * maybe for that we need to change `Taint.Call` to accept a token. *)
+      None
+
+let add_call_to_token_trace ~callee ~var_tokens caller_tokens =
+  (* E.g. (ToReturn) the call to 'bar' in:
+   *
+   *     1 def bar(x):
+   *     2     y = x
+   *     3     return y
+   *     4
+   *     5 def foo():
+   *     6     t = bar(taint)
+   *     7     ...
+   *
+   * would result in this list of tokens (note that is reversed):
+   *
+   *     ["t" @l.6; "y" @l.2; "x" @l.1; "bar" @l.6]
+   *
+   * This is a hack we use because taint traces aren't general enough,
+   * this should be represented with a call trace.
+   *)
+  let call_tokens =
+    (match get_ident_of_callee callee with
+    | None -> []
+    | Some ident -> [ snd ident ])
+    @ List.rev var_tokens
+  in
+  List.rev_append call_tokens caller_tokens
+
+let add_lval_update_to_token_trace ~callee:_TODO lval_tok ~var_tokens
+    caller_tokens =
+  (* E.g. (ToLval) the call to 'bar' in:
+   *
+   *     1 s = set([])
+   *     2
+   *     3 def bar(x):
+   *     4    global s
+   *     5    s.add(x)
+   *     6
+   *     7 def foo():
+   *     8    global s
+   *     9    t = taint
+   *    10    bar(t)
+   *    11    sink(s)
+   *
+   * would result in this list of tokens (note that is reversed):
+   *
+   *     ["s" @l.5; "s" @l.5; "x" @l.3; "s" @l.5; "bar" @l.10; "t" @l.9]
+   *
+   * This is a hack we use because taint traces aren't general enough,
+   * this should be represented with a call trace.
+   *)
+  let call_tokens =
+    (* TODO: Use `get_ident_of_callee callee` to add the callee to the trace. *)
+    lval_tok :: List.rev var_tokens
+  in
+  List.rev_append call_tokens caller_tokens
+
+(*****************************************************************************)
+(* Instatiation *)
+(*****************************************************************************)
+
+let subst_in_precondition inst_var taint =
   let subst taints =
     taints
     |> List.concat_map (fun t ->
            match t.T.orig with
            | Src _ -> [ t ]
            | Var lval -> (
-               match inst_lval lval with
+               match inst_var.inst_lval lval with
                | None -> []
-               | Some (var_taints, _var_shape) -> var_taints |> Taints.elements)
+               | Some (call_taints, _call_shape) ->
+                   call_taints |> Taints.elements)
            | Shape_var lval -> (
-               match inst_lval lval with
+               match inst_var.inst_lval lval with
                | None -> []
-               | Some (_var_taints, var_shape) ->
-                   Shape.gather_all_taints_in_shape var_shape |> Taints.elements
-               )
-           | Control -> inst_ctrl () |> Taints.elements)
+               | Some (_call_taints, call_shape) ->
+                   (* Taint shape-variable, stands for the taints reachable
+                    * through the shape of the 'lval', it's like a delayed
+                    * call to 'Shape.gather_all_taints_in_shape'. *)
+                   Shape.gather_all_taints_in_shape call_shape
+                   |> Taints.elements)
+           | Control -> inst_var.inst_ctrl () |> Taints.elements)
   in
   T.map_preconditions subst taint
 
-let instantiate_taint_var ~inst_lval ~inst_ctrl taint =
+let instantiate_taint_var inst_var taint =
   match taint.T.orig with
   | Src _ -> None
-  | Var lval -> inst_lval lval
+  | Var lval -> inst_var.inst_lval lval
   | Shape_var lval ->
       (* This is just a delayed 'gather_all_taints_in_shape'. *)
       let* taints =
-        inst_lval lval
+        inst_var.inst_lval lval
         |> Option.map (fun (_taints, shape) ->
                Shape.gather_all_taints_in_shape shape)
       in
       Some (taints, Bot)
   | Control ->
       (* 'Control' is pretty much like a taint variable so we handle all together. *)
-      Some (inst_ctrl (), Bot)
+      Some (inst_var.inst_ctrl (), Bot)
 
-let instantiate_taint ~callee ~inst_lval ~inst_ctrl taint =
-  let inst_taint_var taint =
-    instantiate_taint_var ~inst_lval ~inst_ctrl taint
-  in
+let instantiate_taint inst_var inst_trace taint =
+  let inst_taint_var taint = instantiate_taint_var inst_var taint in
   match taint.T.orig with
   | Src src -> (
       let taint =
-        (* Update taint trace.
-         *
-         * E.g. the call to 'bar' in:
-         *
-         *     1 def bar():
-         *     2     x = taint
-         *     3     return x
-         *     4
-         *     5 def foo():
-         *     6     bar()
-         *     7     ...
-         *
-         * would result in this call trace:
-         *
-         *     Call('bar' @l.6, ["x" @l.2], "taint" @l.2)
-         *)
-        match callee with
-        | { IL.e = _; eorig = SameAs orig_callee } ->
-            let call_trace =
-              T.Call (orig_callee, taint.tokens, src.call_trace)
-            in
+        match
+          inst_trace.add_call_to_trace_for_src taint.tokens src.call_trace
+        with
+        | Some call_trace ->
             { T.orig = Src { src with call_trace }; tokens = [] }
-        | __else__ -> taint
+        | None -> taint
       in
-      match subst_in_precondition ~inst_lval ~inst_ctrl taint with
+      match subst_in_precondition inst_var taint with
       | None ->
           (* substitution made preconditon false, so no taint here! *)
           Taints.empty
@@ -126,49 +264,25 @@ let instantiate_taint ~callee ~inst_lval ~inst_ctrl taint =
   | Control -> (
       match inst_taint_var taint with
       | None -> Taints.empty
-      | Some (var_taints, _var_shape) ->
-          (* Update taint trace.
-           *
-           * E.g. the call to 'bar' in:
-           *
-           *     1 def bar(x):
-           *     2     y = x
-           *     3     return y
-           *     4
-           *     5 def foo():
-           *     6     t = bar(taint)
-           *     7     ...
-           *
-           * would result in this list of tokens (note that is reversed):
-           *
-           *     ["t" @l.6; "y" @l.2; "x" @l.1; "bar" @l.6]
-           *
-           * This is a hack we use because taint traces aren't general enough,
-           * this should be represented with a call trace.
-           *)
-          let extra_tokens =
-            (match get_ident_of_callee callee with
-            | None -> []
-            | Some ident -> [ snd ident ])
-            @ List.rev taint.tokens
-          in
-          var_taints
+      | Some (call_taints, _Bot_shape) ->
+          call_taints
           |> Taints.map (fun taint' ->
                  {
                    taint' with
-                   tokens = List.rev_append extra_tokens taint'.tokens;
+                   tokens =
+                     inst_trace.fix_token_trace_for_var ~var_tokens:taint.tokens
+                       taint'.tokens;
                  }))
 
-let instantiate_taints ~callee ~inst_lval ~inst_ctrl taints =
-  taints |> Taints.elements
-  |> List.fold_left
+let instantiate_taints inst_var inst_trace taints =
+  taints |> Taints.to_seq
+  |> Seq.fold_left
        (fun acc taint ->
-         acc
-         |> Taints.union (instantiate_taint ~callee ~inst_lval ~inst_ctrl taint))
+         acc |> Taints.union (instantiate_taint inst_var inst_trace taint))
        Taints.empty
 
-let instantiate_shape ~callee ~inst_lval ~inst_ctrl shape =
-  let inst_taints = instantiate_taints ~callee ~inst_lval ~inst_ctrl in
+let instantiate_shape inst_var inst_trace shape =
+  let inst_taints = instantiate_taints inst_var inst_trace in
   let rec inst_shape = function
     | Bot -> Bot
     | Obj obj ->
@@ -181,7 +295,7 @@ let instantiate_shape ~callee ~inst_lval ~inst_ctrl shape =
         in
         if Fields.is_empty obj then Bot else Obj obj
     | Arg arg -> (
-        match inst_lval (T.lval_of_arg arg) with
+        match inst_var.inst_lval (T.lval_of_arg arg) with
         | Some (_taints, shape) -> shape
         | None ->
             Log.warn (fun m ->
@@ -225,9 +339,9 @@ let find_pos_in_actual_args ?(err_ctx = "???") args_taints fparams :
     (* Here, we take all the named arguments and remove them from the list of parameters.
      *)
     List_.fold_right
-      (fun param acc ->
+      (fun (param : Signature.param) acc ->
         match param with
-        | G.Param { pname = Some (s', _); _ } -> (
+        | P s' -> (
             match SMap.find_opt s' named_arg_map with
             | Some taints ->
                 (* If this parameter is one of our arguments, insert a mapping and then remove it
@@ -236,8 +350,8 @@ let find_pos_in_actual_args ?(err_ctx = "???") args_taints fparams :
                 acc
                 (* Otherwise, it has not been consumed, so keep it in the remaining parameters.*)
             | None -> param :: acc (* Same as above. *))
-        | __else__ -> param :: acc)
-      (Tok.unbracket fparams) []
+        | Other -> param :: acc)
+      fparams []
   in
   let _ =
     (* We then process all of the positional arguments in order of the remaining parameters.
@@ -501,18 +615,30 @@ let taints_of_sig_lval lval_env ~check_lval fparams fun_exp args_exps
     (args_taints : (Taints.t * shape) IL.argument list) (sig_lval : T.lval) =
   match taints_of_lval lval_env fparams fun_exp args_taints sig_lval with
   | Some (taints, shape) -> Some (taints, shape)
-  | None ->
-      (* We want to know what's the taint carried by 'arg_exp.x1. ... .xN'.
-       * TODO: We should not need this when we cover everything with shapes,
-       *   see 'lval_of_sig_lval'.
-       *)
-      let* lval, _obj = lval_of_sig_lval fun_exp fparams args_exps sig_lval in
-      let lval_taints, shape = check_lval lval in
-      let lval_taints =
-        lval_taints
-        |> fix_lval_taints_if_global_or_a_field_of_this_class fun_exp sig_lval
-      in
-      Some (lval_taints, shape)
+  | None -> (
+      match args_exps with
+      | None ->
+          Log.warn (fun m ->
+              m
+                "Cannot find the taint&shape of %s because we lack the actual \
+                 arguments"
+                (T.show_lval sig_lval));
+          None
+      | Some args_exps ->
+          (* We want to know what's the taint carried by 'arg_exp.x1. ... .xN'.
+           * TODO: We should not need this when we cover everything with shapes,
+           *   see 'lval_of_sig_lval'.
+           *)
+          let* lval, _obj =
+            lval_of_sig_lval fun_exp fparams args_exps sig_lval
+          in
+          let lval_taints, shape = check_lval lval in
+          let lval_taints =
+            lval_taints
+            |> fix_lval_taints_if_global_or_a_field_of_this_class fun_exp
+                 sig_lval
+          in
+          Some (lval_taints, shape))
 
 (* This function is consuming the taint signature of a function to determine
    a few things:
@@ -521,26 +647,27 @@ let taints_of_sig_lval lval_env ~check_lval fparams fun_exp args_exps
    2) Are there any effects that occur within the function due to taints being
       input into the function body, from the calling context?
 *)
-let instantiate_function_signature lval_env ~check_lval fparams fun_sig fun_exp
-    fun_eorig args (args_taints : (Taints.t * shape) IL.argument list) : _ =
-  (* This function simply produces the corresponding taints to the
-      given argument, within the body of the function.
-  *)
-  (* Our first pass will be to substitute the args for taints.
-     We can't do this indiscriminately at the beginning, because
-     we might need to use some of the information of the pre-substitution
-     taints and the post-substitution taints, for instance the tokens.
-
-     So we will isolate this as a specific step to be applied as necessary.
-  *)
+let instantiate_function_signature lval_env ~check_lval (fun_sig : Signature.t)
+    ~callee ~(args : _ option)
+    (args_taints : (Taints.t * shape) IL.argument list) : call_effects option =
   let lval_to_taints lval =
+    (* This function simply produces the corresponding taints to the
+        given argument, within the body of the function.
+    *)
+    (* Our first pass will be to substitute the args for taints.
+       We can't do this indiscriminately at the beginning, because
+       we might need to use some of the information of the pre-substitution
+       taints and the post-substitution taints, for instance the tokens.
+
+       So we will isolate this as a specific step to be applied as necessary.
+    *)
     let opt_taints_shape =
-      taints_of_sig_lval lval_env ~check_lval fparams fun_exp args args_taints
-        lval
+      taints_of_sig_lval lval_env ~check_lval fun_sig.params callee args
+        args_taints lval
     in
     Log.debug (fun m ->
-        m ~tags:sigs_tag "Instantiating taint signature of %s: %s -> %s"
-          (Display_IL.string_of_exp fun_exp)
+        m ~tags:sigs_tag "- Instantiating %s: %s -> %s"
+          (Display_IL.string_of_exp callee)
           (T.show_lval lval)
           (match opt_taints_shape with
           | None -> "nothing :/"
@@ -548,34 +675,24 @@ let instantiate_function_signature lval_env ~check_lval fparams fun_sig fun_exp
               spf "%s & %s" (T.show_taints taints) (show_shape shape)));
     opt_taints_shape
   in
+  (* Instantiation helpers *)
   let taints_in_ctrl () = Lval_env.get_control_taints lval_env in
-  let inst_taint_var taint =
-    instantiate_taint_var ~inst_lval:lval_to_taints ~inst_ctrl:taints_in_ctrl
-      taint
+  let inst_var = { inst_lval = lval_to_taints; inst_ctrl = taints_in_ctrl } in
+  let inst_taint_var taint = instantiate_taint_var inst_var taint in
+  let subst_in_precondition = subst_in_precondition inst_var in
+  let inst_trace =
+    {
+      add_call_to_trace_for_src = add_call_to_trace_if_callee_has_eorig ~callee;
+      fix_token_trace_for_var = add_call_to_token_trace ~callee;
+    }
   in
-  let subst_in_precondition =
-    subst_in_precondition ~inst_lval:lval_to_taints ~inst_ctrl:taints_in_ctrl
-  in
-  let process_sig : Effect.t -> _ list = function
+  let inst_taints taints = instantiate_taints inst_var inst_trace taints in
+  let inst_shape shape = instantiate_shape inst_var inst_trace shape in
+  (* Instatiate effects *)
+  let inst_effect : Effect.t -> call_effect list = function
     | Effect.ToReturn { data_taints; data_shape; control_taints; return_tok } ->
-        let inst_taints taints =
-          taints
-          |> List.fold_left
-               (fun return_taints (t : T.taint) ->
-                 let taints' =
-                   (* TODO: Use 'Taint_inst.instantiate_taint' also for 'ToSink' and
-                            'ToLval' cases below. *)
-                   instantiate_taint ~callee:fun_exp ~inst_lval:lval_to_taints
-                     ~inst_ctrl:taints_in_ctrl t
-                 in
-                 return_taints |> Taints.union taints')
-               Taints.empty
-        in
         let data_taints = inst_taints data_taints in
-        let data_shape =
-          instantiate_shape ~callee:fun_exp ~inst_lval:lval_to_taints
-            ~inst_ctrl:taints_in_ctrl data_shape
-        in
+        let data_shape = inst_shape data_shape in
         let control_taints =
           (* No need to instantiate 'control_taints' because control taint variables
            * do not propagate through function calls... BUT instantiation also fixes
@@ -585,12 +702,15 @@ let instantiate_function_signature lval_env ~check_lval fparams fun_sig fun_exp
         if
           Shape.taints_and_shape_are_relevant data_taints data_shape
           || not (Taints.is_empty control_taints)
-        then [ `ToReturn (data_taints, data_shape, control_taints, return_tok) ]
+        then
+          [ ToReturn { data_taints; data_shape; control_taints; return_tok } ]
         else []
-    | Effect.ToSink { taints_with_precondition = taints, _requires; sink; _ } ->
-        let incoming_taints =
+    | Effect.ToSink
+        { taints_with_precondition = taints, requires; sink; merged_env } ->
+        let taints =
           taints
           |> List.concat_map (fun { Effect.taint; sink_trace } ->
+                 (* TODO: Use 'instantiate_taint' here too (note differences wrt the call trace). *)
                  match taint.T.orig with
                  | T.Src _ ->
                      (* Here, we do not modify the call trace or the taint.
@@ -628,20 +748,30 @@ let instantiate_function_signature lval_env ~check_lval fparams fun_sig fun_exp
                  | Shape_var _
                  | Control ->
                      let sink_trace =
-                       T.Call (fun_eorig, taint.tokens, sink_trace)
+                       add_call_to_trace_if_callee_has_eorig ~callee
+                         taint.tokens sink_trace
+                       ||| sink_trace
                      in
-                     let+ var_taints, var_shape = inst_taint_var taint in
+                     let+ call_taints, call_shape = inst_taint_var taint in
                      (* See NOTE(gather-all-taints) *)
-                     let var_taints =
-                       var_taints
+                     let call_taints =
+                       call_taints
                        |> Taints.union
-                            (Shape.gather_all_taints_in_shape var_shape)
+                            (Shape.gather_all_taints_in_shape call_shape)
                      in
-                     Taints.elements var_taints
+                     Taints.elements call_taints
                      |> List_.map (fun x -> { Effect.taint = x; sink_trace }))
         in
-        if List_.null incoming_taints then []
-        else [ `ToSink (incoming_taints, sink) ]
+        if List_.null taints then []
+        else
+          [
+            ToSink
+              {
+                taints_with_precondition = (taints, requires);
+                sink;
+                merged_env;
+              };
+          ]
     | Effect.ToLval (taints, dst_sig_lval) ->
         (* Taints 'taints' go into an argument of the call, by side-effect.
          * Right now this is mainly used to track taint going into specific
@@ -649,42 +779,40 @@ let instantiate_function_signature lval_env ~check_lval fparams fun_sig fun_exp
         let+ dst_lval, tainted_tok =
           (* 'dst_lval' is the actual argument/l-value that corresponds
              * to the formal argument 'dst_sig_lval'. *)
-          lval_of_sig_lval fun_exp fparams args dst_sig_lval
+          match args with
+          | None ->
+              Log.warn (fun m ->
+                  m
+                    "Cannot instantiate '%s' because we lack the actual \
+                     arguments"
+                    (T.show_lval dst_sig_lval));
+              None
+          | Some args ->
+              lval_of_sig_lval callee fun_sig.params args dst_sig_lval
         in
-        taints
-        |> List.concat_map (fun t ->
-               let dst_taints =
-                 match t.T.orig with
-                 | Src src -> (
-                     let call_trace =
-                       T.Call (fun_eorig, t.tokens, src.call_trace)
-                     in
-                     let t =
-                       { Taint.orig = Src { src with call_trace }; tokens = [] }
-                     in
-                     match t |> subst_in_precondition with
-                     | None -> Taints.empty
-                     | Some t -> Taints.singleton t)
-                 | Var _
-                 | Shape_var _ -> (
-                     (* Taint is flowing from one argument to another argument
-                      * (or possibly the callee object). Given the formal poly
-                      * taint 'src_lval', we compute the actual taint in the
-                      * context of this function call. *)
-                     match inst_taint_var t with
-                     | None -> Taints.empty
-                     | Some (res, _TODOshape) ->
-                         res
-                         |> Taints.map (fun taint ->
-                                let tokens =
-                                  t.tokens @ (tainted_tok :: taint.T.tokens)
-                                in
-                                { taint with tokens }))
-                 | Control ->
-                     (* control taints do not propagate to arguments *)
-                     Taints.empty
-               in
-               if Taints.is_empty dst_taints then []
-               else [ `ToLval (dst_taints, dst_lval) ])
+        let taints =
+          taints
+          |> instantiate_taints
+               {
+                 inst_lval = lval_to_taints;
+                 (* Note that control taints do not propagate to l-values. *)
+                 inst_ctrl = (fun _ -> Taints.empty);
+               }
+               {
+                 add_call_to_trace_for_src =
+                   add_call_to_trace_if_callee_has_eorig ~callee;
+                 fix_token_trace_for_var =
+                   add_lval_update_to_token_trace ~callee tainted_tok;
+               }
+        in
+
+        if Taints.is_empty taints then [] else [ ToLval (taints, dst_lval) ]
+  in
+  let call_effects =
+    fun_sig.effects |> Effects.elements |> List.concat_map inst_effect
   in
-  Some (fun_sig |> Signature.elements |> List.concat_map process_sig)
+  Log.debug (fun m ->
+      m ~tags:sigs_tag "Instantiated call to %s: %s"
+        (Display_IL.string_of_exp callee)
+        (show_call_effects call_effects));
+  Some call_effects
diff --git a/src/tainting/Sig_inst.mli b/src/tainting/Sig_inst.mli
index de80dbf7a53a..96e8fbd6ca85 100644
--- a/src/tainting/Sig_inst.mli
+++ b/src/tainting/Sig_inst.mli
@@ -1,22 +1,27 @@
 (** Instantiation of taint signatures *)
 
+(** Like 'Shape_and_sig.Effect.t' but instantiated for a specific call site.
+ * In particular, there is no 'ToSinkInCall' effect, and 'ToLval' effects
+ * refer to specific 'IL.lval's rather than to 'Taint.lval's. *)
+type call_effect =
+  | ToSink of Shape_and_sig.Effect.taints_to_sink
+  | ToReturn of Shape_and_sig.Effect.taints_to_return
+  | ToLval of Taint.taints * IL.lval
+
+type call_effects = call_effect list
+
 val instantiate_function_signature :
   Taint_lval_env.t ->
   check_lval:(IL.lval -> Taint.Taint_set.t * Shape_and_sig.Shape.shape) ->
-  AST_generic.parameters ->
+  (* TODO: 'check_lval' is just a way to avoid a recursive dependency with
+   *   'Dataflow_tainting'. We should not need this when all field-sensitive
+   *   taint tracking happens through shapes. *)
   Shape_and_sig.Signature.t ->
-  IL.exp ->
-  AST_generic.expr ->
-  IL.exp IL.argument list ->
+  callee:IL.exp ->
+  args:IL.exp IL.argument list option (** actual arguments *) ->
   (Taint.Taint_set.t * Shape_and_sig.Shape.shape) IL.argument list ->
-  [ `ToSink of
-    Shape_and_sig.Effect.taint_to_sink_item list * Shape_and_sig.Effect.sink
-  | `ToReturn of
-    Taint.Taint_set.t * Shape_and_sig.Shape.shape * Taint.Taint_set.t * Tok.t
-  | `ToLval of Taint.Taint_set.t * IL.lval ]
-  list
-  option
+  call_effects option
 (** Instantiation is meant to replace the taint and shape variables in the
  * signature of a callee function, with the taints and shapes of the parameters
- * at the call site.
+ * at the call site. It also constructs the call trace.
  *)
diff --git a/src/tainting/Taint.ml b/src/tainting/Taint.ml
index ce36047cbf1e..41774b57a6e1 100644
--- a/src/tainting/Taint.ml
+++ b/src/tainting/Taint.ml
@@ -459,7 +459,7 @@ module Taint_set = struct
                * Otherwise we end up with taint sets where most of the taints are
                * essentially the same. This is probably due to
                * [see note "Taint-tracking via ranges" in Match_tainting_mode],
-               * and not having "Best_sources" [see note "Best matches" in 'Taint_smatch'].
+               * and not having "Best_sources" [see note "Best matches" in 'Taint_spec_match'].
                * TOOD: Revisit ^^^ now we have `exact: true` sources.
                *)
               let ts1' = of_list ts1 in
diff --git a/src/tainting/Taint.mli b/src/tainting/Taint.mli
index 52630059ea4f..64d1fafc4fcc 100644
--- a/src/tainting/Taint.mli
+++ b/src/tainting/Taint.mli
@@ -196,6 +196,7 @@ module Taint_set : sig
   val map : (taint -> taint) -> t -> t
   val iter : (taint -> unit) -> t -> unit
   val fold : (taint -> 'a -> 'a) -> t -> 'a -> 'a
+  val filter : (taint -> bool) -> t -> t
   val of_list : taint list -> t
   val to_seq : t -> taint Seq.t
   val elements : t -> taint list
diff --git a/src/tainting/Taint_shape.ml b/src/tainting/Taint_shape.ml
index 81af34b7a1ec..c60fcd5f4a80 100644
--- a/src/tainting/Taint_shape.ml
+++ b/src/tainting/Taint_shape.ml
@@ -21,6 +21,8 @@ module T = Taint
 module Taints = T.Taint_set
 open Shape_and_sig.Shape
 module Fields = Shape_and_sig.Fields
+module Effects = Shape_and_sig.Effects
+module Signature = Shape_and_sig.Signature
 
 (*********************************************************)
 (* Helpers *)
@@ -46,6 +48,11 @@ let internal_UNSAFE_find_offset_in_obj o obj =
               (T.show_offset o));
         (Oany, obj))
 
+let debug_offset offset =
+  match offset with
+  | [] -> "<NO OFFSET>"
+  | _ :: _ -> offset |> List_.map T.show_offset |> String.concat ""
+
 (*********************************************************)
 (* Misc *)
 (*********************************************************)
@@ -94,11 +101,9 @@ and unify_shape shape1 shape2 =
   match (shape1, shape2) with
   | Bot, shape
   | shape, Bot ->
+      (* 'Bot' acts like a do-not-care. *)
       shape
   | Obj obj1, Obj obj2 -> Obj (unify_obj obj1 obj2)
-  | Arg _, (Obj _ as obj)
-  | (Obj _ as obj), Arg _ ->
-      obj
   | Arg arg1, Arg arg2 ->
       if T.equal_arg arg1 arg2 then shape1
       else (
@@ -117,6 +122,10 @@ and unify_shape shape1 shape2 =
             m "Trying to unify two different arg shapes: %s ~ %s"
               (T.show_arg arg1) (T.show_arg arg2));
         shape1)
+  (* 'Arg' acts like a shape variable. *)
+  | Arg _, (Obj _ as obj)
+  | (Obj _ as obj), Arg _ ->
+      obj
 
 and unify_obj obj1 obj2 =
   (* THINK: Apply taint_MAX_OBJ_FIELDS limit ? *)
@@ -162,12 +171,16 @@ let rec find_in_cell offset cell =
   | [] -> Some cell
   | _ :: _ -> find_in_shape offset shape
 
-and find_in_shape offset = function
+and find_in_shape offset shape =
+  match shape with
   (* offset <> [] *)
   | Bot -> None
   | Obj obj -> find_in_obj offset obj
   | Arg _ ->
       (* TODO: Here we should "refine" the arg shape, it should be an Obj shape. *)
+      Log.warn (fun m ->
+          m "Could not find offset %s in polymorphic shape %s"
+            (debug_offset offset) (show_shape shape));
       None
 
 and find_in_obj (offset : T.offset list) obj =
@@ -222,7 +235,8 @@ let rec update_offset_in_cell ~f offset cell =
   | `Tainted _, (Bot | Obj _ | Arg _) ->
       Some (Cell (xtaint, shape))
 
-and update_offset_in_shape ~f offset = function
+and update_offset_in_shape ~f offset shape =
+  match shape with
   | Bot
   | Arg _ ->
       let shape = Obj Fields.empty in
@@ -317,7 +331,8 @@ let rec clean_cell (offset : T.offset list) cell =
       let shape = clean_shape offset shape in
       Cell (xtaint, shape)
 
-and clean_shape offset = function
+and clean_shape offset shape =
+  match shape with
   | Bot
   | Arg _ ->
       let shape = Obj Fields.empty in
diff --git a/src/tainting/Taint_smatch.ml b/src/tainting/Taint_spec_match.ml
similarity index 100%
rename from src/tainting/Taint_smatch.ml
rename to src/tainting/Taint_spec_match.ml
diff --git a/src/tainting/Taint_smatch.mli b/src/tainting/Taint_spec_match.mli
similarity index 100%
rename from src/tainting/Taint_smatch.mli
rename to src/tainting/Taint_spec_match.mli
diff --git a/src/targeting/Guess_lang.ml b/src/targeting/Guess_lang.ml
index 9b12ed528685..de61bb5974f5 100644
--- a/src/targeting/Guess_lang.ml
+++ b/src/targeting/Guess_lang.ml
@@ -278,6 +278,7 @@ let inspect_file_p (lang : Lang.t) path =
     | Xml
     | Yaml ->
         matches_lang lang
+    | Move_on_sui -> failwith "Unimplemented"
   in
 
   eval test path


Test patch:
diff --git a/cli/src/semgrep/test.py b/cli/src/semgrep/test.py
index 09f6a74efa50..9acc139852b0 100644
--- a/cli/src/semgrep/test.py
+++ b/cli/src/semgrep/test.py
@@ -1,3 +1,5 @@
+# This file is DEPRECATED! Please modify instead osemgrep test in
+# src/osemgrep/cli_test/Test_subcommand.ml
 import collections
 import difflib
 import functools
@@ -41,8 +43,6 @@
 RULEID = "ruleid"
 TODOOK = "todook"
 OK = "ok"
-DEEPOK = "deepok"
-DEEPRULEID = "deepruleid"
 
 EXIT_FAILURE = 2
 
@@ -53,6 +53,8 @@ def _remove_ending_comments(rule: str) -> str:
     return rule
 
 
+# Partial support for pro/deep annotations by just skipping them.
+# Use osemgrep test --pro if you actually want to process those annotations.
 def normalize_rule_ids(line: str) -> Set[str]:
     """
     given a line like `     # ruleid:foobar`
@@ -63,7 +65,12 @@ def normalize_rule_ids(line: str) -> Set[str]:
     _, rules_text = line.strip().split(":", 1)
     rules_text = rules_text.strip()
     # strip out "deepok" and "deepruleid" annotations if they are there to get rule name
-    if rules_text.startswith(DEEPOK) or rules_text.startswith(DEEPRULEID):
+    if (
+        rules_text.startswith("deepok")
+        or rules_text.startswith("prook")
+        or rules_text.startswith("deepruleid")
+        or rules_text.startswith("proruleid")
+    ):
         _, rules_text = rules_text.split(":")
     rules = rules_text.strip().split(",")
     # remove comment ends for non-newline comment syntaxes
diff --git a/cli/tests/default/e2e-other/snapshots/test_pro_rule_skipping/test_pro_rule_skipping/results.json b/cli/tests/default/e2e-other/snapshots/test_pro_rule_skipping/test_pro_rule_skipping/results.json
index 656461fa7eb3..dc7383f24e4d 100644
--- a/cli/tests/default/e2e-other/snapshots/test_pro_rule_skipping/test_pro_rule_skipping/results.json
+++ b/cli/tests/default/e2e-other/snapshots/test_pro_rule_skipping/test_pro_rule_skipping/results.json
@@ -3,7 +3,7 @@
     {
       "code": 0,
       "level": "info",
-      "message": "Missing plugin for rule rules.dummy-apex-rule:\n Missing Semgrep extension needed for parsing Apex target. Try adding `--pro` to your command.",
+      "message": "Missing plugin for rule rules.dummy-apex-rule:\n Missing Semgrep extension needed for parsing Apex target. Try adding `--pro-languages` to your command.",
       "path": "<MASKED>",
       "rule_id": "rules.dummy-apex-rule",
       "type": "Missing plugin"
diff --git a/cli/tests/default/e2e-other/snapshots/test_pro_rule_skipping/test_pro_rule_skipping_no_parsing/results.json b/cli/tests/default/e2e-other/snapshots/test_pro_rule_skipping/test_pro_rule_skipping_no_parsing/results.json
index 927aac285a07..f61ae933fb0d 100644
--- a/cli/tests/default/e2e-other/snapshots/test_pro_rule_skipping/test_pro_rule_skipping_no_parsing/results.json
+++ b/cli/tests/default/e2e-other/snapshots/test_pro_rule_skipping/test_pro_rule_skipping_no_parsing/results.json
@@ -3,7 +3,7 @@
     {
       "code": 0,
       "level": "info",
-      "message": "Missing plugin for rule rules.dummy-apex-rule-no-parsing-string-pattern:\n Missing Semgrep extension needed for parsing Apex target. Try adding `--pro` to your command.",
+      "message": "Missing plugin for rule rules.dummy-apex-rule-no-parsing-string-pattern:\n Missing Semgrep extension needed for parsing Apex target. Try adding `--pro-languages` to your command.",
       "path": "<MASKED>",
       "rule_id": "rules.dummy-apex-rule-no-parsing-string-pattern",
       "type": "Missing plugin"
@@ -11,7 +11,7 @@
     {
       "code": 0,
       "level": "info",
-      "message": "Missing plugin for rule rules.dummy-apex-rule-no-parsing-complex-pattern:\n Missing Semgrep extension needed for parsing Apex target. Try adding `--pro` to your command.",
+      "message": "Missing plugin for rule rules.dummy-apex-rule-no-parsing-complex-pattern:\n Missing Semgrep extension needed for parsing Apex target. Try adding `--pro-languages` to your command.",
       "path": "<MASKED>",
       "rule_id": "rules.dummy-apex-rule-no-parsing-complex-pattern",
       "type": "Missing plugin"
diff --git a/cli/tests/default/e2e-other/snapshots/test_ssc/test_ssc/rulesdependency_awarerust-sca.yaml-dependency_awarerust/results.txt b/cli/tests/default/e2e-other/snapshots/test_ssc/test_ssc/rulesdependency_awarerust-sca.yaml-dependency_awarerust/results.txt
index da3930ea8ed3..6e49acd555e1 100644
--- a/cli/tests/default/e2e-other/snapshots/test_ssc/test_ssc/rulesdependency_awarerust-sca.yaml-dependency_awarerust/results.txt
+++ b/cli/tests/default/e2e-other/snapshots/test_ssc/test_ssc/rulesdependency_awarerust-sca.yaml-dependency_awarerust/results.txt
@@ -46,6 +46,7 @@ SEMGREP_USER_AGENT_APPEND="pytest" SEMGREP_SETTINGS_FILE="<MASKED>" SEMGREP_VERS
                 ]
               },
               "ecosystem": "cargo",
+              "line_number": 6,
               "lockfile_path": "targets/dependency_aware/rust/Cargo.lock",
               "package": "adler",
               "transitivity": "unknown",
diff --git a/cli/tests/default/e2e-other/snapshots/test_ssc/test_ssc/rulesdependency_awarerust-sca.yaml-dependency_awarerust_short_lockfile/results.txt b/cli/tests/default/e2e-other/snapshots/test_ssc/test_ssc/rulesdependency_awarerust-sca.yaml-dependency_awarerust_short_lockfile/results.txt
index ac4b58bd9f0a..89824a5181dc 100644
--- a/cli/tests/default/e2e-other/snapshots/test_ssc/test_ssc/rulesdependency_awarerust-sca.yaml-dependency_awarerust_short_lockfile/results.txt
+++ b/cli/tests/default/e2e-other/snapshots/test_ssc/test_ssc/rulesdependency_awarerust-sca.yaml-dependency_awarerust_short_lockfile/results.txt
@@ -42,6 +42,7 @@ SEMGREP_USER_AGENT_APPEND="pytest" SEMGREP_SETTINGS_FILE="<MASKED>" SEMGREP_VERS
             "found_dependency": {
               "allowed_hashes": {},
               "ecosystem": "cargo",
+              "line_number": 6,
               "lockfile_path": "targets/dependency_aware/rust_short_lockfile/Cargo.lock",
               "package": "adler",
               "transitivity": "unknown",
diff --git a/cli/tests/default/e2e-other/test_ci.py b/cli/tests/default/e2e-other/test_ci.py
index 69dacebf5fd0..8219a211077e 100644
--- a/cli/tests/default/e2e-other/test_ci.py
+++ b/cli/tests/default/e2e-other/test_ci.py
@@ -517,6 +517,17 @@ def _complete_scan_func(semgrep_url: str = "https://semgrep.dev"):
     return _complete_scan_func
 
 
+@pytest.fixture
+def scan_failure_mock_maker(requests_mock, mocked_scan_id):
+    def _scan_failure_func(semgrep_url: str = "https://semgrep.dev"):
+        return requests_mock.post(
+            f"{semgrep_url}/api/agent/scans/{mocked_scan_id}/error",
+            json=json.dumps({"exit_code": 0}),
+        )
+
+    return _scan_failure_func
+
+
 @pytest.fixture
 def mock_ci_api(
     start_scan_mock_maker, upload_results_mock_maker, complete_scan_mock_maker
@@ -1849,18 +1860,24 @@ def test_fail_open_works_when_backend_is_down(
 @pytest.mark.osemfail
 def test_bad_config(
     run_semgrep: RunSemgrep,
+    mocker,
     git_tmp_path_with_commit,
     start_scan_mock_maker,
     complete_scan_mock_maker,
     upload_results_mock_maker,
+    scan_failure_mock_maker,
 ):
     """
-    Test that bad rules has exit code > 1
+    Test that bad rules has exit code > 1 and we notify the app.
     """
 
     start_scan_mock = start_scan_mock_maker("https://semgrep.dev")
     complete_scan_mock = complete_scan_mock_maker("https://semgrep.dev")
     upload_results_mock = upload_results_mock_maker("https://semgrep.dev")
+    scan_failure_mock = scan_failure_mock_maker("https://semgrep.dev")
+
+    # This is the function that notifies the app of the failure.
+    report_failure = mocker.patch.object(ScanHandler, "report_failure")
 
     result = run_semgrep(
         subcommand="ci",
@@ -1872,6 +1889,7 @@ def test_bad_config(
         use_click_runner=True,
     )
     assert "Invalid rule schema" in result.stderr
+    report_failure.assert_called_once()
 
 
 @pytest.mark.parametrize("scan_config", [BAD_CONFIG], ids=["bad_config"])
@@ -1883,15 +1901,23 @@ def test_bad_config_error_handler(
     start_scan_mock_maker,
     complete_scan_mock_maker,
     upload_results_mock_maker,
+    scan_failure_mock_maker,
 ):
     """
     Test that bad rules with --suppres-errors returns exit code 0
+    and we notify the app.
     """
-    mock_send = mocker.spy(ErrorHandler, "send")
+    # This is the function that traps all exceptions at the top level for
+    # all commands.
+    top_level_error_handler = mocker.spy(ErrorHandler, "send")
 
     start_scan_mock = start_scan_mock_maker("https://semgrep.dev")
     complete_scan_mock = complete_scan_mock_maker("https://semgrep.dev")
     upload_results_mock = upload_results_mock_maker("https://semgrep.dev")
+    scan_failure_mock = scan_failure_mock_maker("https://semgrep.dev")
+
+    # This is the function that notifies the app of the failure.
+    report_failure = mocker.patch.object(ScanHandler, "report_failure")
 
     result = run_semgrep(
         subcommand="ci",
@@ -1903,7 +1929,8 @@ def test_bad_config_error_handler(
         use_click_runner=True,
     )
     assert "Invalid rule schema" in result.stderr
-    mock_send.assert_called_once_with(mocker.ANY, 7)
+    top_level_error_handler.assert_called_once_with(mocker.ANY, 7)
+    report_failure.assert_called_once()
 
 
 @pytest.mark.osemfail
diff --git a/cli/tests/default/e2e/snapshots/test_baseline/test_commit_doesnt_exist/error.txt b/cli/tests/default/e2e/snapshots/test_baseline/test_commit_doesnt_exist/error.txt
index 89631a9ed5ea..91c06c8c5eaa 100644
--- a/cli/tests/default/e2e/snapshots/test_baseline/test_commit_doesnt_exist/error.txt
+++ b/cli/tests/default/e2e/snapshots/test_baseline/test_commit_doesnt_exist/error.txt
@@ -8,6 +8,8 @@ Failed to run 'git cat-file -e 12345'. Possible reasons:
 
 - the git binary is not available
 - the current working directory is not a git repository
+- the baseline commit is not a parent of the current commit
+    (if you are running through semgrep-app, check if you are setting `SEMGREP_BRANCH` or `SEMGREP_BASELINE_COMMIT` properly)
 - the current working directory is not marked as safe
     (fix with `git config --global --add safe.directory $(pwd)`)
 
diff --git a/cli/tests/default/e2e/snapshots/test_misc/test_show_supported_languages/results.txt b/cli/tests/default/e2e/snapshots/test_misc/test_show_supported_languages/results.txt
index fcf4f2123c35..06818637e9c2 100644
--- a/cli/tests/default/e2e/snapshots/test_misc/test_show_supported_languages/results.txt
+++ b/cli/tests/default/e2e/snapshots/test_misc/test_show_supported_languages/results.txt
@@ -1,1 +1,1 @@
-supported languages are: apex, bash, c, c#, c++, cairo, circom, clojure, cpp, csharp, dart, docker, dockerfile, elixir, ex, generic, go, golang, hack, hcl, html, java, javascript, js, json, jsonnet, julia, kotlin, kt, lisp, lua, move_on_aptos, none, ocaml, php, promql, proto, proto3, protobuf, py, python, python2, python3, ql, r, regex, ruby, rust, scala, scheme, sh, sol, solidity, swift, terraform, tf, ts, typescript, vue, xml, yaml
+supported languages are: apex, bash, c, c#, c++, cairo, circom, clojure, cpp, csharp, dart, docker, dockerfile, elixir, ex, generic, go, golang, hack, hcl, html, java, javascript, js, json, jsonnet, julia, kotlin, kt, lisp, lua, move_on_aptos, move_on_sui, none, ocaml, php, promql, proto, proto3, protobuf, py, python, python2, python3, ql, r, regex, ruby, rust, scala, scheme, sh, sol, solidity, swift, terraform, tf, ts, typescript, vue, xml, yaml
diff --git a/cli/tests/default/e2e/snapshots/test_rule_parser/test_rule_parser__failure__error_messages/badlanguage/error-in-color.txt b/cli/tests/default/e2e/snapshots/test_rule_parser/test_rule_parser__failure__error_messages/badlanguage/error-in-color.txt
index f516e022a785..edbde9ff2c86 100644
--- a/cli/tests/default/e2e/snapshots/test_rule_parser/test_rule_parser__failure__error_messages/badlanguage/error-in-color.txt
+++ b/cli/tests/default/e2e/snapshots/test_rule_parser/test_rule_parser__failure__error_messages/badlanguage/error-in-color.txt
@@ -5,7 +5,7 @@
 [94m[22m[24m  | [0m               [31m[22m[24m^^^^^^^^^^[0m
 [94m[22m[24m8 | [0m    severity: WARNING
 
-[31m[22m[24munsupported language: intercal. supported languages are: apex, bash, c, c#, c++, cairo, circom, clojure, cpp, csharp, dart, docker, dockerfile, elixir, ex, generic, go, golang, hack, hcl, html, java, javascript, js, json, jsonnet, julia, kotlin, kt, lisp, lua, move_on_aptos, none, ocaml, php, promql, proto, proto3, protobuf, py, python, python2, python3, ql, r, regex, ruby, rust, scala, scheme, sh, sol, solidity, swift, terraform, tf, ts, typescript, vue, xml, yaml
+[31m[22m[24munsupported language: intercal. supported languages are: apex, bash, c, c#, c++, cairo, circom, clojure, cpp, csharp, dart, docker, dockerfile, elixir, ex, generic, go, golang, hack, hcl, html, java, javascript, js, json, jsonnet, julia, kotlin, kt, lisp, lua, move_on_aptos, move_on_sui, none, ocaml, php, promql, proto, proto3, protobuf, py, python, python2, python3, ql, r, regex, ruby, rust, scala, scheme, sh, sol, solidity, swift, terraform, tf, ts, typescript, vue, xml, yaml
 
 You may need to update your version of Semgrep, if you are on an old version that does not yet support this language.[0m
 
diff --git a/cli/tests/default/e2e/snapshots/test_rule_parser/test_rule_parser__failure__error_messages/badlanguage/error.json b/cli/tests/default/e2e/snapshots/test_rule_parser/test_rule_parser__failure__error_messages/badlanguage/error.json
index d28533a65fb4..d72e6d965187 100644
--- a/cli/tests/default/e2e/snapshots/test_rule_parser/test_rule_parser__failure__error_messages/badlanguage/error.json
+++ b/cli/tests/default/e2e/snapshots/test_rule_parser/test_rule_parser__failure__error_messages/badlanguage/error.json
@@ -3,7 +3,7 @@
     {
       "code": 8,
       "level": "error",
-      "long_msg": "unsupported language: intercal. supported languages are: apex, bash, c, c#, c++, cairo, circom, clojure, cpp, csharp, dart, docker, dockerfile, elixir, ex, generic, go, golang, hack, hcl, html, java, javascript, js, json, jsonnet, julia, kotlin, kt, lisp, lua, move_on_aptos, none, ocaml, php, promql, proto, proto3, protobuf, py, python, python2, python3, ql, r, regex, ruby, rust, scala, scheme, sh, sol, solidity, swift, terraform, tf, ts, typescript, vue, xml, yaml\n\nYou may need to update your version of Semgrep, if you are on an old version that does not yet support this language.",
+      "long_msg": "unsupported language: intercal. supported languages are: apex, bash, c, c#, c++, cairo, circom, clojure, cpp, csharp, dart, docker, dockerfile, elixir, ex, generic, go, golang, hack, hcl, html, java, javascript, js, json, jsonnet, julia, kotlin, kt, lisp, lua, move_on_aptos, move_on_sui, none, ocaml, php, promql, proto, proto3, protobuf, py, python, python2, python3, ql, r, regex, ruby, rust, scala, scheme, sh, sol, solidity, swift, terraform, tf, ts, typescript, vue, xml, yaml\n\nYou may need to update your version of Semgrep, if you are on an old version that does not yet support this language.",
       "short_msg": "invalid language: intercal",
       "spans": [
         {
diff --git a/cli/tests/default/unit/test_clean_project_url.py b/cli/tests/default/unit/test_clean_project_url.py
index eb6572ae2495..19e4546fa117 100644
--- a/cli/tests/default/unit/test_clean_project_url.py
+++ b/cli/tests/default/unit/test_clean_project_url.py
@@ -1,6 +1,9 @@
+from unittest.mock import patch
+
 import pytest
 
 from semgrep.git import clean_project_url
+from semgrep.git import get_project_url
 
 
 @pytest.mark.quick
@@ -11,3 +14,10 @@ def test_git_url_clean():
         )
         == "https://gitlab.company.world/app/test-case.git"
     )
+
+
+@pytest.mark.quick
+@patch("semgrep.git.clean_project_url")
+def test_get_project_url(patched_clean_project_url):
+    get_project_url()
+    patched_clean_project_url.assert_called_once()
diff --git a/src/engine/tests/Test_compare_matches.ml b/src/engine/tests/Test_compare_matches.ml
index a854a7c7b1bf..58c0f613e77c 100644
--- a/src/engine/tests/Test_compare_matches.ml
+++ b/src/engine/tests/Test_compare_matches.ml
@@ -78,7 +78,7 @@ let compare_actual_to_expected ~to_location actual_findings
   let num_errors = false_negatives + false_positives in
   let msg =
     spf
-      "The test failed to find exactly the expected findings: %i false \
+      "The test failed to produce exactly the expected findings: %i false \
        negative%s, %i false positive%s."
       false_negatives (plural false_negatives) false_positives
       (plural false_positives)
diff --git a/src/engine/tests/Test_dataflow_tainting.ml b/src/engine/tests/Test_dataflow_tainting.ml
index 97c2190212a6..e8aa8dff5ffa 100644
--- a/src/engine/tests/Test_dataflow_tainting.ml
+++ b/src/engine/tests/Test_dataflow_tainting.ml
@@ -64,26 +64,26 @@ let test_dfg_tainting rules_file file =
   let rule = List_.hd_exn "unexpected empty list" taint_rules in
   UCommon.pr2 "Tainting";
   UCommon.pr2 "========";
-  let handle_findings _ _ _ = () in
+  let handle_effects _ _ = () in
   let xconf = Match_env.default_xconfig in
   let xconf = Match_env.adjust_xconfig_with_rule_options xconf rule.options in
   (* this won't cache anything. but that's fine, we don't need it
      for test purposes.
   *)
-  let tbl = Match_tainting_mode.mk_specialized_formula_cache [] in
-  let config, debug_taint, _exps =
-    Match_tainting_mode.taint_config_of_rule ~per_file_formula_cache:tbl xconf
-      !!file (ast, []) rule handle_findings
+  let tbl = Formula_cache.mk_specialized_formula_cache [] in
+  let config, spec_matches, _exps =
+    Match_taint_spec.taint_config_of_rule ~per_file_formula_cache:tbl xconf
+      !!file (ast, []) rule handle_effects
   in
   UCommon.pr2 "\nSources";
   UCommon.pr2 "-------";
-  pr2_ranges file (debug_taint.sources |> List_.map fst);
+  pr2_ranges file (spec_matches.sources |> List_.map fst);
   UCommon.pr2 "\nSanitizers";
   UCommon.pr2 "----------";
-  pr2_ranges file debug_taint.sanitizers;
+  pr2_ranges file (spec_matches.sanitizers |> List_.map fst);
   UCommon.pr2 "\nSinks";
   UCommon.pr2 "-----";
-  pr2_ranges file (debug_taint.sinks |> List_.map fst);
+  pr2_ranges file (spec_matches.sinks |> List_.map fst);
   let v =
     object
       inherit [_] AST_generic.iter_no_id_info as super
diff --git a/src/engine/tests/Test_engine.ml b/src/engine/tests/Test_engine.ml
index e10388c65ef2..6cce02c03795 100644
--- a/src/engine/tests/Test_engine.ml
+++ b/src/engine/tests/Test_engine.ml
@@ -296,6 +296,10 @@ let find_rule_files roots =
 (* Entry point *)
 (*****************************************************************************)
 
+(* TODO: do not filter_map, instead fail when we can't parse a rule
+ * (or wait that we switch to osemgrep test for our own test infra in which
+ * case this whole file will be deleted)
+ *)
 let collect_tests ?(get_xlang = single_xlang_from_rules) (xs : Fpath.t list) =
   xs |> find_rule_files
   |> List_.filter_map (fun rule_file ->
diff --git a/src/osemgrep/cli_test/Test_annotation.ml b/src/osemgrep/cli_test/Test_annotation.ml
index 7d0f78d2019a..f9d02102f917 100644
--- a/src/osemgrep/cli_test/Test_annotation.ml
+++ b/src/osemgrep/cli_test/Test_annotation.ml
@@ -1,11 +1,38 @@
+(* Yoann Padioleau
+ *
+ * Copyright (C) 2024 Semgrep Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public License
+ * version 2.1 as published by the Free Software Foundation, with the
+ * special exception on linking described in file LICENSE.
+ *
+ * This library is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the file
+ * LICENSE for more details.
+ *)
 open Common
 open Fpath_.Operators
 
 (*****************************************************************************)
 (* Prelude *)
 (*****************************************************************************)
-(* Module to represent and parse test annotations.
- * See https://semgrep.dev/docs/writing-rules/testing-rules/
+(* Module to represent and parse semgrep test annotations.
+ * See https://semgrep.dev/docs/writing-rules/testing-rules/ for more info.
+ *
+ * Note that our docs currently mention just the OSS test annotations but not
+ * the pro one (e.g., 'proruleid:', 'deepruleid:').
+ * See tests/intrafile/README for more info on the pro/deep annotations.
+ *
+ * Note that if a finding is expected in OSS but not in Pro, you may need to
+ * combine two annotations, e.g. `ruleid: prook: test` would mean that the
+ * finding is expected in OSS but in Pro we expect no finding, which could be
+ * due to inter-procedural analysis being able to spot sanitization via a
+ * third-function.
+ *
+ * TODO: update https://semgrep.dev/docs/writing-rules/testing-rules and add
+ * doc for the pro/deep test annotations and remove tests/intrafile/README.
  *)
 
 (*****************************************************************************)
@@ -15,23 +42,48 @@ open Fpath_.Operators
 type kind =
   (* The good one, should be reported (TP) *)
   | Ruleid
-  (* Those should *not* be reported (TN) *)
-  | Ok
   (* Should be reported but are not because of current engine limitations (FN) *)
   | Todoruleid
   (* Are reported but should not (FP) *)
   | Todook
+  (* Those should *not* be reported (TN)
+   * The 'ok:' is not that useful (it's mostly a comment) and actually
+   * complicates some code during parsing (see the _no_ok regexps below), but
+   * the `prook:` and `deepok:` are useful to "negate" a preceding ruleid: when
+   * a legitimate finding in semgrep OSS is actually considered a FP for the
+   * pro engine and should not be reported.
+   *)
+  | Ok
 [@@deriving show]
 
-(* following the conventions used in the annotations themselves with
+(* Here we follow the conventions used in the annotations themselves with
  * proruleid: and deepruleid: (see tests/intrafile/README).
  * alt: CoreScan | ProScan | DeepScan
  * less: factorize with the other engine types
  *)
 type engine = OSS | Pro | Deep [@@deriving show]
 
-(* ex: "#ruleid: lang.ocaml.do-not-use-lisp-map" *)
-type t = kind * engine * Rule_ID.t [@@deriving show]
+(* ex: "#ruleid: lang.ocaml.do-not-use-lisp-map"
+ * but also "ruleid: prook: lang.ocaml.do-not-use-lisp-map".
+ *
+ * Note that 'ruleid:' implies 'proruleid:' and 'deepruleid:' so you don't need
+ * to repeat those annotations. You usually need multiple kind/engine
+ * prefix when one engine TP would be another engine TN (e.g., 'ruleid: prook:')
+ * Here are a few implicit rules:
+ *  - ruleid: > proruleid: > deepruleid:
+ *  - prook: > deepok: ?
+ *)
+type t = {
+  kind : kind;
+  engine : engine;
+  (* e.g., a ruleid: prook: x
+   * alt: call the field negators?
+   *)
+  others : (kind * engine) list;
+  (* alt: ids: Rule_ID.t list; (instead we return a list of annots) *)
+  id : Rule_ID.t;
+}
+[@@deriving show]
 
 (* starts at 1 *)
 type linenb = int
@@ -41,6 +93,7 @@ let prefilter_annotation_regexp = ".*\\(ruleid\\|ok\\|todoruleid\\|todook\\):.*"
 
 (* removing ok as it could be valid code (as in `ok: foo` in JS)
  * alt: choose an annotation for ok: that would be less ambiguous
+ * alt: get rid of ok:, just care about prook: and deepok:
  *)
 let prefilter_annotation_regexp_no_ok =
   ".*\\(ruleid\\|todoruleid\\|todook\\):.*"
@@ -66,6 +119,14 @@ let (comment_syntaxes : (string * string option) list) =
 (* Helpers *)
 (*****************************************************************************)
 
+let kind_of_string (str : string) : kind =
+  match str with
+  | "ruleid" -> Ruleid
+  | "ok" -> Ok
+  | "todoruleid" -> Todoruleid
+  | "todook" -> Todook
+  | s -> failwith (spf "not a valid annotation: %s" s)
+
 let remove_enclosing_comment_opt (str : string) : string option =
   comment_syntaxes
   |> List.find_map (fun (prefix, suffixopt) ->
@@ -103,23 +164,46 @@ let () =
       test_remove "<!-- foobar -->" (Some " foobar ");
       ())
 
+(* returns the rest of the string too *)
+let parse_kind_and_engine_opt (s : string) : (kind * engine * string) option =
+  let engine, s =
+    match s with
+    | _ when s =~ "^pro\\(.*\\)" -> (Pro, Common.matched1 s)
+    | _ when s =~ "^deep\\(.*\\)" -> (Deep, Common.matched1 s)
+    | _ -> (OSS, s)
+  in
+  if s =~ annotation_regexp then
+    let kind_str, s = Common.matched2 s in
+    let kind = kind_of_string kind_str in
+    let s = String.trim s in
+    Some (kind, engine, s)
+  else None
+
+(* TODO: could check for bad combinations such as
+ * - ruleid: deepruleid: proruleid:  (redundant)
+ * - todoruleid: deepruleid: (probably just need deepruleid:)
+ *)
+let parse_kinds_and_engines_opt (s : string) :
+    (kind * engine * (kind * engine) list * string) option =
+  let* kind, engine, s = parse_kind_and_engine_opt s in
+  let rec aux annots s =
+    match parse_kind_and_engine_opt s with
+    | None -> Some (kind, engine, List.rev annots, s)
+    | Some (kind2, engine2, s) -> aux ((kind2, engine2) :: annots) s
+  in
+  aux [] s
+
 (*****************************************************************************)
 (* Parsing *)
 (*****************************************************************************)
 
-let annotation_kind_of_string (str : string) : kind =
-  match str with
-  | "ruleid" -> Ruleid
-  | "ok" -> Ok
-  | "todoruleid" -> Todoruleid
-  | "todook" -> Todook
-  | s -> failwith (spf "not a valid annotation: %s" s)
-
 (* This does a few things:
  *  - check comments: #, //, ( *, <--
+ *  - support pro/deep annotations
+ *  - support possible deepok: or prook: following the ruleid: (to negate
+ *    the ruleid when running a ProScan or DeepScan)
  *  - support multiple ruleids separated by commas
- *  - support possible leading deepok:
- *  - TODO? support pro/deep annotations?
+ * alt: use parser combinators instead of those regexps/trims/Str.string_xxx
  *)
 let annotations_of_string (orig_str : string) (file : Fpath.t) (idx : linenb) :
     annotations =
@@ -132,55 +216,41 @@ let annotations_of_string (orig_str : string) (file : Fpath.t) (idx : linenb) :
     let res = remove_enclosing_comment_opt s in
     match res with
     | None ->
+        (* some Javascript code has valid code such as { ok: true } that is not
+         * a semgrep annotation hence the use of a no_ok prefilter below
+         *)
         if s =~ prefilter_annotation_regexp_no_ok then
           Logs.err (fun m ->
               m "annotation without leading comment: %s" orig_str)
-          (* some Javascript code has valid code such as { ok: true } that is not
-           * a semgrep annotation *)
         else
           Logs.debug (fun m ->
               m "skipping %s, actually not an annotation" orig_str);
         []
-    | Some s ->
+    | Some s -> (
         (* " ruleid: foo.bar " *)
         let s = String.trim s in
         (* "ruleid: foo.bar" *)
-        let engine, s =
-          match s with
-          | _ when s =~ "^pro\\(.*\\)" -> (Pro, Common.matched1 s)
-          | _ when s =~ "^deep\\(.*\\)" -> (Deep, Common.matched1 s)
-          | _ -> (OSS, s)
-        in
-        if s =~ annotation_regexp then
-          let kind_str, ids_str = Common.matched2 s in
-          let kind = annotation_kind_of_string kind_str in
-          let s = String.trim ids_str in
-          let s =
-            (* indicate that no finding is expected in interfile analysis *)
-            let prefix = "deepok:" in
-            if String.starts_with ~prefix s then
-              Str.string_after s (String.length prefix)
-            else ids_str
-          in
-          let xs =
-            Str.split_delim (Str.regexp "[ \t]*,[ \t]*") s
-            |> List_.map String.trim
-          in
-          xs
-          |> List_.filter_map (fun id_str ->
-                 match Rule_ID.of_string_opt id_str with
-                 | Some id -> Some ((kind, engine, id), idx)
-                 | None ->
-                     Logs.warn (fun m ->
-                         m
-                           "malformed rule ID '%s' (%s) skipping this \
-                            annotation"
-                           id_str error_context);
-                     None)
-        else (
-          Logs.warn (fun m ->
-              m "could not parse annotation: %s (%s)" orig_str error_context);
-          [])
+        match parse_kinds_and_engines_opt s with
+        | Some (kind, engine, others, s) ->
+            let xs =
+              Str.split_delim (Str.regexp "[ \t]*,[ \t]*") s
+              |> List_.map String.trim
+            in
+            xs
+            |> List_.filter_map (fun id_str ->
+                   match Rule_ID.of_string_opt id_str with
+                   | Some id -> Some ({ kind; engine; others; id }, idx)
+                   | None ->
+                       Logs.warn (fun m ->
+                           m
+                             "malformed rule ID '%s' (%s) skipping this \
+                              annotation"
+                             id_str error_context);
+                       None)
+        | None ->
+            Logs.warn (fun m ->
+                m "could not parse annotation: %s (%s)" orig_str error_context);
+            [])
   else []
 
 (* Note that this returns the line of the annotation itself. In practice,
@@ -206,19 +276,27 @@ let () =
             (spf "Annotations didn't match, got %s, expected %s"
                (Dumper.dump xs) (Dumper.dump expected))
       in
-      test "// ruleid: foo.bar"
-        [ (Ruleid, OSS, Rule_ID.of_string_exn "foo.bar") ];
+      let rule_id s = Rule_ID.of_string_exn s in
+      test "// ruleid: foo"
+        [ { kind = Ruleid; engine = OSS; others = []; id = rule_id "foo" } ];
       test "// ruleid: foo, bar"
         [
-          (Ruleid, OSS, Rule_ID.of_string_exn "foo");
-          (Ruleid, OSS, Rule_ID.of_string_exn "bar");
+          { kind = Ruleid; engine = OSS; others = []; id = rule_id "foo" };
+          { kind = Ruleid; engine = OSS; others = []; id = rule_id "bar" };
         ];
       test "<!-- ruleid: foo-bar -->"
-        [ (Ruleid, OSS, Rule_ID.of_string_exn "foo-bar") ];
+        [ { kind = Ruleid; engine = OSS; others = []; id = rule_id "foo-bar" } ];
       (* the ok: does not mean it's an annot; it's regular (JS) code *)
       test "return res.send({ok: true})" [];
-      test "// ruleid: deepok: foo.deep"
-        [ (Ruleid, OSS, Rule_ID.of_string_exn "foo.deep") ];
+      test "// ruleid: deepok: foo"
+        [
+          {
+            kind = Ruleid;
+            engine = OSS;
+            others = [ (Ok, Deep) ];
+            id = rule_id "foo";
+          };
+        ];
       ())
 
 (*****************************************************************************)
@@ -231,7 +309,7 @@ let () =
 let group_positive_annotations (annots : annotations) :
     (Rule_ID.t, linenb list) Assoc.t =
   annots
-  |> List_.filter_map (fun ((kind, engine, id), line) ->
+  |> List_.filter_map (fun ({ kind; engine; others = _; id }, line) ->
          match kind with
          | Ruleid
          | Todook ->
@@ -250,7 +328,7 @@ let group_positive_annotations (annots : annotations) :
 let filter_todook (annots : annotations) (xs : linenb list) : linenb list =
   let (todooks : linenb Set_.t) =
     annots
-    |> List_.filter_map (fun ((kind, _engine, _id), line) ->
+    |> List_.filter_map (fun ({ kind; _ }, line) ->
            match kind with
            (* + 1 because the expected/reported is the line after the annotation *)
            | Todook -> Some (line + 1)
diff --git a/src/osemgrep/cli_test/Test_annotation.mli b/src/osemgrep/cli_test/Test_annotation.mli
index 0959890049a5..f18c09f9580e 100644
--- a/src/osemgrep/cli_test/Test_annotation.mli
+++ b/src/osemgrep/cli_test/Test_annotation.mli
@@ -1,18 +1,31 @@
 type kind =
   (* The good one, should be reported (TP) *)
   | Ruleid
-  (* Those should *not* be reported (TN) *)
-  | Ok
   (* Should be reported but are not because of current engine limitations (FN) *)
   | Todoruleid
   (* Are reported but should not (FP) *)
   | Todook
+  (* Those should *not* be reported (TN) *)
+  | Ok
 [@@deriving show]
 
 type engine = OSS | Pro | Deep [@@deriving show]
 
-(* ex: "#ruleid: lang.ocaml.do-not-use-lisp-map" *)
-type t = kind * engine * Rule_ID.t [@@deriving show]
+(* ex: "#ruleid: lang.ocaml.do-not-use-lisp-map"
+ * but also "ruleid: prook: lang.ocaml.do-not-use-lisp-map".
+ *
+ * Note that 'ruleid:' implies 'proruleid:' and 'deepruleid:' so you don't need
+ * to repeat those annotations. You usually need multiple kind/engine
+ * prefix when one engine TP would be another engine FP (e.g., 'ruleid: prook:')
+ *)
+type t = {
+  kind : kind;
+  engine : engine;
+  (* e.g., to deal with multiple annots as in 'ruleid: prook: x' *)
+  others : (kind * engine) list;
+  id : Rule_ID.t;
+}
+[@@deriving show]
 
 (* starts at 1 *)
 type linenb = int
diff --git a/src/osemgrep/cli_test/Test_subcommand.ml b/src/osemgrep/cli_test/Test_subcommand.ml
index 07838d450135..3cc05444e0da 100644
--- a/src/osemgrep/cli_test/Test_subcommand.ml
+++ b/src/osemgrep/cli_test/Test_subcommand.ml
@@ -626,7 +626,8 @@ let run_tests (caps : Core_scan.caps) (conf : Test_CLI.conf) (tests : tests)
                       (* TODO: filter different annots if pro engine *)
                       let annots =
                         annots
-                        |> List.filter (fun ((_, engine, _), _) ->
+                        |> List.filter
+                             (fun (Test_annotation.{ engine; _ }, _) ->
                                match engine with
                                | A.OSS -> true
                                | A.Pro
diff --git a/src/osemgrep/language_server/Test_LS_e2e.ml b/src/osemgrep/language_server/Test_LS_e2e.ml
index b1f1926a9502..bb62542621f1 100644
--- a/src/osemgrep/language_server/Test_LS_e2e.ml
+++ b/src/osemgrep/language_server/Test_LS_e2e.ml
@@ -194,7 +194,9 @@ let send_map (type a) (info : server_info) packet (f : Packet.t -> a) :
 (*****************************************************************************)
 
 let send_request info request =
-  let id = Uuidm.v `V4 |> Uuidm.to_string in
+  let id =
+    Uuidm.v4_gen (Stdlib.Random.State.make_self_init ()) () |> Uuidm.to_string
+  in
   let packet = Packet.Request (CR.to_jsonrpc_request request (`String id)) in
   send_map info packet
 
diff --git a/tests/parsing/dockerfile/todo/numeric-user.dockerfile b/tests/parsing/dockerfile/numeric-user.dockerfile
similarity index 100%
rename from tests/parsing/dockerfile/todo/numeric-user.dockerfile
rename to tests/parsing/dockerfile/numeric-user.dockerfile
diff --git a/tests/parsing/php/TODO/enum.php b/tests/parsing/php/enum.php
similarity index 100%
rename from tests/parsing/php/TODO/enum.php
rename to tests/parsing/php/enum.php
diff --git a/tests/parsing/swift/todo/statements.swift b/tests/parsing/swift/statements.swift
similarity index 100%
rename from tests/parsing/swift/todo/statements.swift
rename to tests/parsing/swift/statements.swift
diff --git a/tests/parsing/c/parsing_missing/missing-semicolon.c b/tests/parsing_missing/c/missing-semicolon.c
similarity index 100%
rename from tests/parsing/c/parsing_missing/missing-semicolon.c
rename to tests/parsing_missing/c/missing-semicolon.c
diff --git a/tests/parsing/cpp/parsing_missing/missing-semicolon.cpp b/tests/parsing_missing/cpp/missing-semicolon.cpp
similarity index 100%
rename from tests/parsing/cpp/parsing_missing/missing-semicolon.cpp
rename to tests/parsing_missing/cpp/missing-semicolon.cpp
diff --git a/tests/parsing/cpp/parsing_partial/empty_case_body.cpp b/tests/parsing_partial/cpp/empty_case_body.cpp
similarity index 100%
rename from tests/parsing/cpp/parsing_partial/empty_case_body.cpp
rename to tests/parsing_partial/cpp/empty_case_body.cpp
diff --git a/tests/parsing/cpp/parsing_partial/expecting_ident_param.cpp b/tests/parsing_partial/cpp/expecting_ident_param.cpp
similarity index 100%
rename from tests/parsing/cpp/parsing_partial/expecting_ident_param.cpp
rename to tests/parsing_partial/cpp/expecting_ident_param.cpp
diff --git a/tests/parsing/cpp/parsing_partial/funcdef_without_params.cpp b/tests/parsing_partial/cpp/funcdef_without_params.cpp
similarity index 100%
rename from tests/parsing/cpp/parsing_partial/funcdef_without_params.cpp
rename to tests/parsing_partial/cpp/funcdef_without_params.cpp
diff --git a/tests/parsing/cpp/parsing_partial/funcdef_without_params2.cpp b/tests/parsing_partial/cpp/funcdef_without_params2.cpp
similarity index 100%
rename from tests/parsing/cpp/parsing_partial/funcdef_without_params2.cpp
rename to tests/parsing_partial/cpp/funcdef_without_params2.cpp
diff --git a/tests/parsing/cpp/parsing_partial/operator_templated.cpp b/tests/parsing_partial/cpp/operator_templated.cpp
similarity index 100%
rename from tests/parsing/cpp/parsing_partial/operator_templated.cpp
rename to tests/parsing_partial/cpp/operator_templated.cpp
diff --git a/tests/parsing/cpp/parsing_partial/operator_templated_partial.cpp b/tests/parsing_partial/cpp/operator_templated_partial.cpp
similarity index 100%
rename from tests/parsing/cpp/parsing_partial/operator_templated_partial.cpp
rename to tests/parsing_partial/cpp/operator_templated_partial.cpp
diff --git a/tests/parsing/lua/todo/builtins.lua b/tests/parsing_todo/lua/builtins.lua
similarity index 100%
rename from tests/parsing/lua/todo/builtins.lua
rename to tests/parsing_todo/lua/builtins.lua
diff --git a/tests/parsing/protobuf/todo/error_todo b/tests/parsing_todo/protobuf/error_todo.proto
similarity index 100%
rename from tests/parsing/protobuf/todo/error_todo
rename to tests/parsing_todo/protobuf/error_todo.proto
diff --git a/tests/rules/neg_op_lit_equiv.c b/tests/rules/neg_op_lit_equiv.c
new file mode 100644
index 000000000000..71c01729aebf
--- /dev/null
+++ b/tests/rules/neg_op_lit_equiv.c
@@ -0,0 +1,27 @@
+#include <limits.h>
+#include <stdlib.h>
+
+void func_one(int i_a) {
+  //ruleid: c-string-equality
+  int result = -i_a;
+}
+
+void func_three(int i_a) {
+  int result;
+  if (i_a == INT_MIN) {
+    exit(EXIT_FAILURE);
+  } else {
+    //ok: c-string-equality
+    result = -i_a;
+  }
+}
+
+void func_four(int i_a) {
+  int result;
+  if (i_a == -123123) {
+    exit(EXIT_FAILURE);
+  } else {
+    //ok: c-string-equality
+    result = -i_a;
+  }
+}
diff --git a/tests/rules/neg_op_lit_equiv.yaml b/tests/rules/neg_op_lit_equiv.yaml
new file mode 100644
index 000000000000..75b198cebdcc
--- /dev/null
+++ b/tests/rules/neg_op_lit_equiv.yaml
@@ -0,0 +1,23 @@
+rules:
+  - id: c-string-equality
+    languages:
+      - c
+    severity: ERROR
+    message: Using == on char* performs pointer comparison, use strcmp instead
+    patterns:
+      - pattern-either:
+          - pattern: <... -(int $NUMBER) ...>;
+      - pattern-not-inside: |-
+          if (<... $I_A == INT_MIN ...>) {
+            ...
+          } else {
+            ...
+            ...
+          }
+      - pattern-not-inside: |-
+          if (<... $I_A == -123123 ...>) {
+            ...
+          } else {
+            ...
+            ...
+          }
diff --git a/tests/rules/taint_param_default1.py b/tests/rules/taint_param_default1.py
new file mode 100644
index 000000000000..bb3905f880e8
--- /dev/null
+++ b/tests/rules/taint_param_default1.py
@@ -0,0 +1,3 @@
+#ruleid: test
+def test(x=sink(taint)):
+  pass
diff --git a/tests/rules/taint_param_default1.yaml b/tests/rules/taint_param_default1.yaml
new file mode 100644
index 000000000000..2abed5fb1c1a
--- /dev/null
+++ b/tests/rules/taint_param_default1.yaml
@@ -0,0 +1,10 @@
+rules:
+  - id: test
+    message: Semgrep found a match
+    languages: [python]
+    severity: WARNING
+    mode: taint
+    pattern-sources:
+      - pattern: taint
+    pattern-sinks:
+      - pattern: sink(...)
diff --git a/tests/semgrep-rules b/tests/semgrep-rules
index 81e40c5acfbc..0da2dcec5660 160000
--- a/tests/semgrep-rules
+++ b/tests/semgrep-rules
@@ -1,1 +1,1 @@
-Subproject commit 81e40c5acfbccba211ca9b8857c7011ac57741da
+Subproject commit 0da2dcec566011e8fcc1dbf7e76387b3972c2f36
diff --git a/tests/snapshots/semgrep-core/745a04125f26/stdout b/tests/snapshots/semgrep-core/745a04125f26/stdout
index 94454b14c8a7..776084c27ffb 100644
--- a/tests/snapshots/semgrep-core/745a04125f26/stdout
+++ b/tests/snapshots/semgrep-core/745a04125f26/stdout
@@ -1,2 +1,2 @@
 Snapshot for Test_show_subcommand.test_supported_languages.(fun)
-supported languages are: apex, bash, c, c#, c++, cairo, circom, clojure, cpp, csharp, dart, docker, dockerfile, elixir, ex, generic, go, golang, hack, hcl, html, java, javascript, js, json, jsonnet, julia, kotlin, kt, lisp, lua, move_on_aptos, none, ocaml, php, promql, proto, proto3, protobuf, py, python, python2, python3, ql, r, regex, ruby, rust, scala, scheme, sh, sol, solidity, swift, terraform, tf, ts, typescript, vue, xml, yaml
+supported languages are: apex, bash, c, c#, c++, cairo, circom, clojure, cpp, csharp, dart, docker, dockerfile, elixir, ex, generic, go, golang, hack, hcl, html, java, javascript, js, json, jsonnet, julia, kotlin, kt, lisp, lua, move_on_aptos, move_on_sui, none, ocaml, php, promql, proto, proto3, protobuf, py, python, python2, python3, ql, r, regex, ruby, rust, scala, scheme, sh, sol, solidity, swift, terraform, tf, ts, typescript, vue, xml, yaml