internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mabel-dev/opteryx

Issue:
✨ enable SQLAlchemy logging
### Thanks for stopping by to let us know something could be better!

**Is your feature request related to a problem? Please describe.** _A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]_

**Describe the solution you'd like** _A clear and concise description of what you want to happen._

**Describe alternatives you've considered** _A clear and concise description of any alternative solutions or features you've considered._

**Additional context** _Add any other context or screenshots about the feature request here._



Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 340
+__build__ = 341
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/connectors/sql_connector.py b/opteryx/connectors/sql_connector.py
--- a/opteryx/connectors/sql_connector.py
+++ b/opteryx/connectors/sql_connector.py
@@ -14,6 +14,7 @@
 The SQL Connector downloads data from remote servers and converts them
 to pyarrow tables so they can be processed as per any other data source.
 """
+import os
 import time
 from decimal import Decimal
 from typing import Any
@@ -40,6 +41,8 @@
 from opteryx.managers.expression import NodeType
 from opteryx.third_party.query_builder import Query
 
+DEBUG_ENABLED = os.environ.get("OPTERYX_DEBUG") is not None
+
 
 def _handle_operand(operand: Node, parameters: dict) -> Tuple[Any, dict]:
     if operand.node_type == NodeType.IDENTIFIER:
@@ -103,7 +106,7 @@ def __init__(self, *args, connection: str = None, engine=None, **kwargs):
 
         # create the SqlAlchemy engine
         if engine is None:
-            self._engine = create_engine(connection, poolclass=NullPool)
+            self._engine = create_engine(connection, poolclass=NullPool, echo=DEBUG_ENABLED)
         else:
             self._engine = engine
 
@@ -168,7 +171,7 @@ def read_dataset(  # type:ignore
             # DEBUG: log ("READ DATASET\n", str(query_builder))
             # DEBUG: log ("PARAMETERS\n", parameters)
             # Execution Options allows us to handle datasets larger than memory
-            result = conn.execution_options(stream_results=True, max_row_buffer=10000).execute(
+            result = conn.execution_options(stream_results=True, max_row_buffer=25000).execute(
                 text(str(query_builder)), parameters=parameters
             )
 


Test patch:
Repository: mabel-dev/opteryx

Issue:
✨ Detect use of `USING` for execute and raise error
`EXECUTE function USING params`


Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 343
+__build__ = 344
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/components/logical_planner/logical_planner.py b/opteryx/components/logical_planner/logical_planner.py
--- a/opteryx/components/logical_planner/logical_planner.py
+++ b/opteryx/components/logical_planner/logical_planner.py
@@ -737,6 +737,12 @@ def plan_execute_query(statement) -> LogicalPlan:
     from opteryx.third_party import sqloxide
     from opteryx.utils import sql
 
+    # the parser allows USING, but we want EXECUTE function (parmeters)
+    if statement["Execute"].get("using"):
+        raise UnsupportedSyntaxError(
+            "EXECUTE does not support USING syntax, please provide parameters in parenthesis."
+        )
+
     statement_name = statement["Execute"]["name"]["value"]
     parameters = [
         logical_planner_builders.build(p["Value"]) for p in statement["Execute"]["parameters"]


Test patch:
Repository: mabel-dev/opteryx

Issue:
✨ LIST_CONTAINS_ANY implementation needs performance tuning
### Thanks for stopping by to let us know something could be better!

**Is your feature request related to a problem? Please describe.** _A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]_

**Describe the solution you'd like** _A clear and concise description of what you want to happen._

**Describe alternatives you've considered** _A clear and concise description of any alternative solutions or features you've considered._

**Additional context** _Add any other context or screenshots about the feature request here._



Discussion:
just under 1m rows per second (1.6m in 1.9s), was about 4.3s - room for improvement - but less than half the time is significant.

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 348
+__build__ = 349
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/functions/__init__.py b/opteryx/functions/__init__.py
--- a/opteryx/functions/__init__.py
+++ b/opteryx/functions/__init__.py
@@ -335,7 +335,7 @@ def select_values(boolean_arrays, value_arrays):
     "GET": _get,
     "GET_STRING": _get_string,
     "LIST_CONTAINS": _iterate_double_parameter(other_functions.list_contains),
-    "LIST_CONTAINS_ANY": _iterate_double_parameter(other_functions.list_contains_any),
+    "LIST_CONTAINS_ANY": other_functions.list_contains_any,
     "LIST_CONTAINS_ALL": _iterate_double_parameter(other_functions.list_contains_all),
     "SEARCH": other_functions.search,
     "COALESCE": _coalesce,
diff --git a/opteryx/functions/other_functions.py b/opteryx/functions/other_functions.py
--- a/opteryx/functions/other_functions.py
+++ b/opteryx/functions/other_functions.py
@@ -26,13 +26,24 @@ def list_contains(array, item):
     return item in set(array)
 
 
-def list_contains_any(array, items):
+def list_contains_any(array: numpy.ndarray, items: numpy.ndarray) -> numpy.ndarray:
     """
-    does array contain any of the items in items
+    Check if any element in each sub-array of 'array' is present in 'items'.
+
+    Parameters:
+        array (numpy.ndarray): A 1D array where each element is an iterable (e.g., list, set).
+        items (numpy.ndarray): An array of items to check against each sub-array in 'array'.
+
+    Returns:
+        numpy.ndarray: A boolean array where each element indicates whether any item
+                       from 'items' is found in the corresponding sub-array of 'array'.
     """
-    if array is None:
-        return False
-    return set(array).intersection(items) != set()
+    items_set = set(items[0])
+    res = numpy.empty(array.size, dtype=bool)
+    for i, test_set in enumerate(array):
+        # Using not to correctly capture overlap (isdisjoint is True when no common elements)
+        res[i] = not items_set.isdisjoint(test_set) if test_set is not None else False
+    return res
 
 
 def list_contains_all(array, items):


Test patch:
Repository: mabel-dev/opteryx

Issue:
✨ CROSS JOIN UNNEST is still too slow (1 million row per second)
### Thanks for stopping by to let us know something could be better!

**Is your feature request related to a problem? Please describe.** _A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]_

**Describe the solution you'd like** _A clear and concise description of what you want to happen._

**Describe alternatives you've considered** _A clear and concise description of any alternative solutions or features you've considered._

**Additional context** _Add any other context or screenshots about the feature request here._



Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 374
+__build__ = 375
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/connectors/gcp_firestore_connector.py b/opteryx/connectors/gcp_firestore_connector.py
--- a/opteryx/connectors/gcp_firestore_connector.py
+++ b/opteryx/connectors/gcp_firestore_connector.py
@@ -18,7 +18,6 @@
 from opteryx import config
 from opteryx.connectors.base.base_connector import INITIAL_CHUNK_SIZE
 from opteryx.connectors.base.base_connector import BaseConnector
-from opteryx.connectors.base.base_connector import DatasetReader
 from opteryx.exceptions import DatasetNotFoundError
 from opteryx.exceptions import MissingDependencyError
 from opteryx.exceptions import UnmetRequirementError
diff --git a/opteryx/operators/cross_join_node.py b/opteryx/operators/cross_join_node.py
--- a/opteryx/operators/cross_join_node.py
+++ b/opteryx/operators/cross_join_node.py
@@ -31,8 +31,8 @@
 from opteryx.models import QueryProperties
 from opteryx.operators import BasePlanNode
 
-INTERNAL_BATCH_SIZE: int = 10  # config
-MAX_JOIN_SIZE: int = 500  # config
+INTERNAL_BATCH_SIZE: int = 5000  # config
+MAX_JOIN_SIZE: int = 1000  # config
 
 
 def _cross_join_unnest_column(


Test patch:
Repository: mabel-dev/opteryx

Issue:
🪲 random appears to be evaluated once
was seeing what happens for opteryx for https://buttondown.email/jaffray/archive/the-case-of-a-curious-sql-query/

~~~sql
SELECT count(*) 
FROM GENERATE_SERIES(1000) AS one_thousand 
CROSS JOIN GENERATE_SERIES(1000) AS one_thousand_b 
WHERE random() < 0.5
~~~

~~~
AFTER COST OPTIMIZATION
└─ EXIT
   └─ PROJECT (COUNT(*))
      └─ AGGREGATE (COUNT(*))
         └─ FILTER (False)
            └─ CROSS JOIN
               ├─ GENERATE SERIES (1000) AS one_thousand
               └─ GENERATE SERIES (1000) AS one_thousand_b
~~~


Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 430
+__build__ = 432
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/components/cost_based_optimizer/strategies/constant_folding.py b/opteryx/components/cost_based_optimizer/strategies/constant_folding.py
--- a/opteryx/components/cost_based_optimizer/strategies/constant_folding.py
+++ b/opteryx/components/cost_based_optimizer/strategies/constant_folding.py
@@ -57,6 +57,11 @@ def build_literal_node(value: Any, root: Node):
 
 def fold_constants(root: Node) -> Node:
     identifiers = get_all_nodes_of_type(root, (NodeType.IDENTIFIER, NodeType.WILDCARD))
+    functions = get_all_nodes_of_type(root, (NodeType.FUNCTION,))
+
+    if any(func.value in {"RANDOM", "RAND", "NORMAL", "RANDOM_STRING"} for func in functions):
+        return root
+
     if len(identifiers) == 0:
         table = no_table_data.read()
         try:


Test patch:
diff --git a/tests/plan_optimization/test_fold_all_constants.py b/tests/plan_optimization/test_fold_all_constants.py
new file mode 100644
--- /dev/null
+++ b/tests/plan_optimization/test_fold_all_constants.py
@@ -0,0 +1,24 @@
+import os
+import sys
+
+sys.path.insert(1, os.path.join(sys.path[0], "../.."))
+
+import numpy
+import opteryx
+
+
+def test_we_dont_fold_random():
+
+    SQL = "SELECT random() AS r FROM GENERATE_SERIES(5000) AS g"
+    df = opteryx.query(SQL)["r"]
+    p25, p50, p75, p95, p99 = numpy.percentile(df, [25, 50, 75, 95, 99])
+
+    assert 0.4 < numpy.mean(df) < 0.6
+    assert 0.2 < p25 < 0.3, p25
+    assert 0.7 < p75 < 0.8, p75
+
+
+if __name__ == "__main__":  # pragma: no cover
+    from tests.tools import run_tests
+
+    run_tests()
Repository: mabel-dev/opteryx

Issue:
✨ setting memcache should gracefully fail - and not try again
### Thanks for stopping by to let us know something could be better!

**Is your feature request related to a problem? Please describe.** _A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]_

**Describe the solution you'd like** _A clear and concise description of what you want to happen._

**Describe alternatives you've considered** _A clear and concise description of any alternative solutions or features you've considered._

**Additional context** _Add any other context or screenshots about the feature request here._



Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 459
+__build__ = 461
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/managers/cache/memcached.py b/opteryx/managers/cache/memcached.py
--- a/opteryx/managers/cache/memcached.py
+++ b/opteryx/managers/cache/memcached.py
@@ -12,7 +12,8 @@
 """
 This implements an interface to Memcached
 
-If we have 10 failures in a row, stop trying to use the cache.
+If we have 10 failures in a row, stop trying to use the cache. We have some
+scenarios where we assume the remote server is down and stop immediately.
 """
 
 import os
@@ -116,4 +117,8 @@ def get(self, key: bytes) -> Union[bytes, None]:
 
     def set(self, key: bytes, value: bytes) -> None:
         if self._consecutive_failures < MAXIMUM_CONSECUTIVE_FAILURES:
-            self._server.set(key, value)
+            try:
+                self._server.set(key, value)
+            except:
+                # if we fail to set, stop trying
+                self._consecutive_failures = MAXIMUM_CONSECUTIVE_FAILURES
diff --git a/opteryx/utils/file_decoders.py b/opteryx/utils/file_decoders.py
--- a/opteryx/utils/file_decoders.py
+++ b/opteryx/utils/file_decoders.py
@@ -160,7 +160,9 @@ def parquet_decoder(buffer, projection: List = None, selection=None, just_schema
         selected_columns = []
 
     # Read the parquet table with the optimized column list and selection filters
-    table = parquet.read_table(stream, columns=selected_columns, pre_buffer=False, filters=_select)
+    table = parquet.read_table(
+        stream, columns=selected_columns, pre_buffer=False, filters=_select, use_threads=False
+    )
     if selection:
         table = filter_records(selection, table)
     if projection == []:


Test patch:
Repository: mabel-dev/opteryx

Issue:
🪲 pyarrow reads are not threadsafe
https://github.com/apache/arrow/issues/33189


Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 462
+__build__ = 464
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/operators/async_scanner_node.py b/opteryx/operators/async_scanner_node.py
--- a/opteryx/operators/async_scanner_node.py
+++ b/opteryx/operators/async_scanner_node.py
@@ -145,7 +145,9 @@ def execute(self) -> Generator:
             blob_name, reference = item
 
             decoder = get_decoder(blob_name)
-            blob_bytes = self.pool.read_and_release(reference)
+            # This pool is being used by async processes in another thread, using
+            # zero copy versions occassionally results in data getting corrupted
+            blob_bytes = self.pool.read_and_release(reference, zero_copy=False)
             try:
                 decoded = decoder(blob_bytes, projection=self.columns, selection=self.predicates)
                 num_rows, _, morsel = decoded
diff --git a/opteryx/shared/memory_pool.py b/opteryx/shared/memory_pool.py
--- a/opteryx/shared/memory_pool.py
+++ b/opteryx/shared/memory_pool.py
@@ -21,6 +21,7 @@
 import asyncio
 from multiprocessing import Lock
 from typing import Dict
+from typing import Union
 
 from orso.tools import random_int
 
@@ -177,7 +178,7 @@ def commit(self, data: bytes) -> int:
             self.commits += 1
             return ref_id
 
-    def read(self, ref_id: int) -> bytes:
+    def read(self, ref_id: int, zero_copy: bool = True) -> Union[bytes, memoryview]:
         """
         We're using an optimistic locking strategy where we do not acquire
         a lock, perform the read and then check that the metadata hasn't changed
@@ -199,7 +200,9 @@ def read(self, ref_id: int) -> bytes:
                 self.read_locks += 1
                 segment = self.used_segments[ref_id]
                 view = memoryview(self.pool)[segment.start : segment.start + segment.length]
-        return view
+        if zero_copy:
+            return view
+        return bytes(view)
 
     def release(self, ref_id: int):
         """
@@ -212,7 +215,7 @@ def release(self, ref_id: int):
             segment = self.used_segments.pop(ref_id)
             self.free_segments.append(segment)
 
-    def read_and_release(self, ref_id: int):
+    def read_and_release(self, ref_id: int, zero_copy: bool = True) -> Union[bytes, memoryview]:
         """
         Combine two steps together, we lock everytime here
         """
@@ -225,7 +228,9 @@ def read_and_release(self, ref_id: int):
             segment = self.used_segments.pop(ref_id)
             view = memoryview(self.pool)[segment.start : segment.start + segment.length]
             self.free_segments.append(segment)
-            return view
+            if zero_copy:
+                return view
+            return bytes(view)
 
     @property
     def stats(self) -> dict:


Test patch:
Repository: mabel-dev/opteryx

Issue:
🪲 mabel partitioning bug
### Thank you for taking the time to report a problem with Opteryx.
_To help us to respond to your request we ask that you try to provide the below detail about the bug._

**Describe the bug** _A clear and specific description of what the bug is. What the error, incorrect or unexpected behaviour was._


**Expected behaviour** _A clear and concise description of what you expected to happen._


**Sample Code/Statement** _If you can, please submit the SQL statement or Python code snippet, or a representative example using the sample datasets._

~~~sql

~~~

**Additional context** _Add any other context about the problem here, for example what you have done to try to diagnose or workaround the problem._



Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 486
+__build__ = 487
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/managers/schemes/mabel_partitions.py b/opteryx/managers/schemes/mabel_partitions.py
--- a/opteryx/managers/schemes/mabel_partitions.py
+++ b/opteryx/managers/schemes/mabel_partitions.py
@@ -98,7 +98,6 @@ def _inner(*, timestamp):
 
             control_blobs: List[str] = []
             data_blobs: List[str] = []
-            segments = set()
             as_ats = set()
             hour_blobs: List[str] = []
 
@@ -112,19 +111,19 @@ def _inner(*, timestamp):
                     if hour_label in blob:
                         hour_blobs.append(blob)
 
+            if hour_blobs:
+                data_blobs = hour_blobs
+
+            for blob in blob_names:
                 # Collect segments
                 if by_label in blob:
-                    segments.add(extract_prefix(blob, "by_"))
+                    segment = extract_prefix(blob, "by_")
+                    if segment != "by_hour":
+                        raise UnsupportedSegementationError(dataset=prefix, segments=[segment])
 
                 if as_at_label in blob:
                     as_ats.add(extract_prefix(blob, "as_at_"))
 
-            if hour_blobs:
-                data_blobs = hour_blobs
-
-            if segments - {"by_hour", None}:
-                raise UnsupportedSegementationError(dataset=prefix, segments=segments)
-
             as_at = None
             as_at_list = sorted(as_ats)
 


Test patch:
Repository: mabel-dev/opteryx

Issue:
🪲 Very Large Morsels (1m rows+) Fail some transforms
Testing with the ClickBench dataset, some situations have failures


Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 512
+__build__ = 514
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/managers/expression/__init__.py b/opteryx/managers/expression/__init__.py
--- a/opteryx/managers/expression/__init__.py
+++ b/opteryx/managers/expression/__init__.py
@@ -310,7 +310,6 @@ def evaluate(expression: Node, table: Table, context: Optional[ExecutionContext]
         context = ExecutionContext()
 
     result = _inner_evaluate(root=expression, table=table, context=context)
-
     if not isinstance(result, (pyarrow.Array, numpy.ndarray)):
         result = numpy.array(result)
     return result
@@ -360,6 +359,9 @@ def evaluate_and_append(expressions, table: Table):
 
         if should_evaluate(statement):
             new_column = evaluate_statement(statement, table, context)
+
+            if isinstance(new_column, pyarrow.ChunkedArray):
+                new_column = new_column.combine_chunks()
             table = table.append_column(statement.schema_column.identity, new_column)
             for identity in context.identities:
                 if identity not in table.column_names:


Test patch:
Repository: mabel-dev/opteryx

Issue:
🪲 [CI] MyPy test failure
### Thank you for taking the time to report a problem with Opteryx.
_To help us to respond to your request we ask that you try to provide the below detail about the bug._

**Describe the bug** _A clear and specific description of what the bug is. What the error, incorrect or unexpected behaviour was._


**Expected behaviour** _A clear and concise description of what you expected to happen._


**Sample Code/Statement** _If you can, please submit the SQL statement or Python code snippet, or a representative example using the sample datasets._

~~~sql

~~~

**Additional context** _Add any other context about the problem here, for example what you have done to try to diagnose or workaround the problem._



Discussion:

Resolving patch:
diff --git a/opteryx/__version__.py b/opteryx/__version__.py
--- a/opteryx/__version__.py
+++ b/opteryx/__version__.py
@@ -1,4 +1,4 @@
-__build__ = 522
+__build__ = 523
 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/opteryx/planner/views/__init__.py b/opteryx/planner/views/__init__.py
--- a/opteryx/planner/views/__init__.py
+++ b/opteryx/planner/views/__init__.py
@@ -12,8 +12,7 @@
 
 import orjson
 
-from opteryx.managers.expression import NodeType
-from opteryx.third_party.travers import Graph
+from opteryx.planner.logical_planner import LogicalPlan
 
 
 def _load_views():
@@ -32,7 +31,7 @@ def is_view(view_name: str) -> bool:
     return view_name in VIEWS
 
 
-def view_as_plan(view_name: str) -> Graph:
+def view_as_plan(view_name: str) -> LogicalPlan:
     from opteryx.planner.logical_planner import do_logical_planning_phase
     from opteryx.third_party import sqloxide
     from opteryx.utils.sql import clean_statement


Test patch: