Repository: mathesar-foundation/mathesar

Issue:
docker compose failing at pip install due to missing dependencies
## Description
<!-- A clear and concise description of what the bug is. -->
When running `docker compose -f docker-compose.yml -f docker-compose.dev.yml up dev-service
`
```
=> [dev-service internal] load .dockerignore                                                                                                      0.0s
 => => transferring context: 143B                                                                                                                  0.0s
 => [dev-service internal] load build definition from Dockerfile                                                                                   0.0s
 => => transferring dockerfile: 2.45kB                                                                                                             0.0s
 => [dev-service internal] load metadata for docker.io/library/python:3.9-buster                                                                   0.7s
 => [dev-service  1/13] FROM docker.io/library/python:3.9-buster@sha256:bae5b428ebf32d01a902718b0a58874cbf33d7a4b6a65b7cd7b21d48b0d2e2f1           0.0s
 => [dev-service internal] load build context                                                                                                      0.1s
 => => transferring context: 194.31kB                                                                                                              0.1s
 => CACHED [dev-service  2/13] RUN set -eux;                                                                                                       0.0s
 => CACHED [dev-service  3/13] RUN mkdir -p /etc/apt/keyrings;                                                                                     0.0s
 => CACHED [dev-service  4/13] RUN curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - ;     echo "deb http://apt.postgresql.  0.0s
 => CACHED [dev-service  5/13] RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/node  0.0s
 => CACHED [dev-service  6/13] RUN apt-get update &&     apt-get install -y --no-install-recommends         sudo         ca-certificates           0.0s
 => CACHED [dev-service  7/13] RUN localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8                                    0.0s
 => CACHED [dev-service  8/13] RUN apt-get update &&     apt-get install -y --no-install-recommends         postgresql-15 postgresql-client-15 po  0.0s
 => CACHED [dev-service  9/13] WORKDIR /code/                                                                                                      0.0s
 => CACHED [dev-service 10/13] COPY requirements* ./                                                                                               0.0s
 => ERROR [dev-service 11/13] RUN pip install --no-cache-dir -r requirements-dev.txt --force-reinstall sqlalchemy-filters                          1.6s
------
 > [dev-service 11/13] RUN pip install --no-cache-dir -r requirements-dev.txt --force-reinstall sqlalchemy-filters:
0.826 Collecting sqlalchemy_filters
0.826   Cloning https://github.com/mathesar-foundation/sqlalchemy-filters (to revision models_to_tables) to /tmp/pip-install-461qq5s5/sqlalchemy-filters_d80dc47043234f738a4aa95eaa753ae5
0.828   Running command git clone --filter=blob:none --quiet https://github.com/mathesar-foundation/sqlalchemy-filters /tmp/pip-install-461qq5s5/sqlalchemy-filters_d80dc47043234f738a4aa95eaa753ae5
1.224   fatal: could not read Username for 'https://github.com': No such device or address
1.243   error: subprocess-exited-with-error
1.243   
1.243   × git clone --filter=blob:none --quiet https://github.com/mathesar-foundation/sqlalchemy-filters /tmp/pip-install-461qq5s5/sqlalchemy-filters_d80dc47043234f738a4aa95eaa753ae5 did not run successfully.
1.243   │ exit code: 128
1.243   ╰─> See above for output.
1.243   
1.243   note: This error originates from a subprocess, and is likely not a problem with pip.
1.246 error: subprocess-exited-with-error
1.246 
1.246 × git clone --filter=blob:none --quiet https://github.com/mathesar-foundation/sqlalchemy-filters /tmp/pip-install-461qq5s5/sqlalchemy-filters_d80dc47043234f738a4aa95eaa753ae5 did not run successfully.
1.246 │ exit code: 128
1.246 ╰─> See above for output.
1.246 
1.246 note: This error originates from a subprocess, and is likely not a problem with pip.
1.541 
1.541 [notice] A new release of pip is available: 23.0.1 -> 23.3.1
1.541 [notice] To update, run: pip install --upgrade pip
------
failed to solve: process "/bin/sh -c pip install --no-cache-dir -r ${PYTHON_REQUIREMENTS} --force-reinstall sqlalchemy-filters" did not complete successfully: exit code: 1
```
It is failing because in the requirements .txt there is :
https://github.com/mathesar-foundation/mathesar/blob/08743e6aac39aa3497269e9bb6b4e9b0216f85b9/requirements.txt#L26

and there does not exist any https://github.com/mathesar-foundation/sqlalchemy-filters
(there is though https://github.com/centerofci-archive/sqlalchemy-filters)

This is probably because of recent changes to the org name.
## Expected behavior
<!-- A clear and concise description of what you expected to happen. -->
Dependencies should be resolved with no issue.

## To Reproduce
<!-- How can we recreate this bug? Please try to provide a Minimal, Complete, and Verifiable (http://stackoverflow.com/help/mcve) example if code-related. -->
Clone a fresh repo or even try to rebuild the image by
`docker compose -f docker-compose.yml -f docker-compose.dev.yml up dev-service --force-recreate --build dev-service
`

## Environment
 - OS: macOS 13.3.1; 
 - Browser: Chrome
 - Other info:

## Additional context
<!-- Add any other context about the problem or screenshots here. -->



Discussion:
It could be solved in two ways:
1. Fork a version of `sqlalchemy-filters` in mathesar-foundation from centerofci-archive
2. Point the github repo to centerofci-archive in requirements.txt
Thanks for reporting this and performing some preliminary analysis, @dikwickley

@kgodey I've assigned this to you since I figure you would be the best person to make the decisions required to fix this.

I've not yet attempted to reproduce this, but assuming it _is_ reproducible, this looks to be a high-priority issue since it has the potential to affect the workflow of all devs.
We are no longer using `sqlalchemy_filters` (that's why it was moved to the`centerofci-archive` org), so removing that dependency entirely should fix this. @dikwickley do you want to take a crack at this and confirm that it's fixed?
@kgodey sure!
Great thanks, I've assigned it to you!
@kgodey i removed the package from requirement.txt and it installed fine. Next I ran tests for backend and here are few which failed
```
=========================== short test summary info ============================
FAILED mathesar/tests/api/test_record_api.py::test_record_search - NotImpleme...
FAILED mathesar/tests/api/test_table_api.py::test_create_table_and_normalize_json_data_file
FAILED mathesar/tests/api/test_table_api.py::test_table_patch_columns_two_name_changes
FAILED mathesar/tests/api/test_table_settings_api.py::test_update_table_settings_permission[db_editor_client_factory-200]
FAILED db/tests/records/operations/test_relevance.py::test_rank_and_filter_rows
==== 5 failed, 2328 passed, 10 skipped, 1124 warnings in 210.02s (0:03:30) =====
```

You did say that we are no longer using `sqlalchemy_filters` but it is still used here:
https://github.com/mathesar-foundation/mathesar/blob/08743e6aac39aa3497269e9bb6b4e9b0216f85b9/db/records/operations/relevance.py#L2

Which is ultimately used in
https://github.com/mathesar-foundation/mathesar/blob/08743e6aac39aa3497269e9bb6b4e9b0216f85b9/db/records/operations/select.py#L21-L52

Till we are using the [apply_transformations_deprecated](https://github.com/mathesar-foundation/mathesar/blob/develop/db/transforms/operations/apply.py#L20) i don't think we should get rid of `sqlalchemy_filters`
Ugh, I did not know we were still using it in one place, thanks for finding this @dikwickley. 

@mathemancer or @Anish9901, is this usage of the package easy to remove? Otherwise, I'll move the `sqlalchemy_filters` package back to the main organization and that should fix it.
The package should be easy to remove @kgodey,
@dikwickley try to rewrite this using sqlalchemy's `order_by`:
https://github.com/mathesar-foundation/mathesar/blob/cf92053259d8864b4f492be20993e077fcb5085d/db/records/operations/relevance.py#L21-L24

Resolving patch:
diff --git a/db/records/operations/relevance.py b/db/records/operations/relevance.py
--- a/db/records/operations/relevance.py
+++ b/db/records/operations/relevance.py
@@ -1,5 +1,4 @@
-from sqlalchemy import case, select
-from sqlalchemy_filters import apply_sort
+from sqlalchemy import case, select, desc
 from db.types import categories
 from db.types.operations.convert import get_db_type_enum_from_class
 
@@ -18,10 +17,7 @@ def get_rank_and_filter_rows_query(relation, parameters_dict, limit=10):
     parameters given in parameters_dict.
     """
     rank_cte = _get_scored_selectable(relation, parameters_dict)
-    filtered_ordered_cte = apply_sort(
-        select(rank_cte).where(rank_cte.columns[SCORE_COL] > 0),
-        {'field': SCORE_COL, 'direction': 'desc'}
-    ).cte()
+    filtered_ordered_cte = select(rank_cte).where(rank_cte.columns[SCORE_COL] > 0).order_by(desc(SCORE_COL)).cte()
     return select(
         *[filtered_ordered_cte.columns[c] for c in [col.name for col in relation.columns]]
     ).limit(limit)


Test patch: