Repository: internetarchive/openlibrary

Issue:
pre-commit: Add Python linting tool ruff
Add https://beta.ruff.rs to our pre-commit as a replacement for `flake8`, `isort`, `pyupgrade`, etc. but written in Rust instead of Python. It can lint the CPython codebase from scratch in 0.29 seconds.

Adopting Ruff should greatly accelerate our pre-commits and our pre-commit.ci jobs. We will run the new and old tools in parallel for a sprint or two to verify the coverage before we completely drop the old tools.


Discussion:
% `ruff check --select=ALL  --statistics .`
```
warning: `one-blank-line-before-class` (D203) and `no-blank-line-before-class` (D211) are incompatible. Ignoring `one-blank-line-before-class`.
warning: `multi-line-summary-first-line` (D212) and `multi-line-summary-second-line` (D213) are incompatible. Ignoring `multi-line-summary-second-line`.
   19	E402   	Module level import not at top of file
   32	E722   	Do not use bare `except`
    4	E741   	Ambiguous variable name: `l`
  116	F401   	`json` imported but unused
   61	F841   	Local variable `account` is assigned to but never used
    1	PLR0402	Use `from os import path` in lieu of alias
    1	PLR1701	Merge these isinstance calls: `isinstance(val, (luqum.tree.Phrase, luqum.tree.Word))`
  219	PLR2004	Magic value used in comparison, consider replacing 100 with a constant variable
   10	PLW0602	Using global for `client` but no assignment is done
   19	PLR0911	Too many return statements (10/6)
   51	PLR0913	Too many arguments to function call (8/5)
   28	PLR0912	Too many branches (19/12)
   17	PLR0915	Too many statements (56/50)
   87	A001   	Variable `hash` is shadowing a python builtin
   73	A002   	Argument `hash` is shadowing a python builtin
   18	A003   	Class attribute `set` is shadowing a python builtin
   41	B007   	Loop control variable `i` not used within loop body
    1	B008   	Do not perform function call `datetime` in argument defaults
   13	B009   	Do not call `getattr` with a constant attribute value. It is not any safer than normal property access.
    1	B010   	Do not call `setattr` with a constant attribute value. It is not any safer than normal property access.
    1	B011   	Do not `assert False` (`python -O` removes these calls), raise `AssertionError()`
   20	B015   	Pointless comparison. This comparison does nothing but waste CPU instructions. Either prepend `assert` or remove it.
   10	B023   	Function definition does not bind loop variable `lang`
   32	B904   	Within an except clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling
   19	B905   	`zip()` without an explicit `strict=` parameter
   30	BLE001 	Do not catch blind exception: `Exception`
    1	C400   	Unnecessary generator (rewrite as a `list` comprehension)
    1	C406   	Unnecessary `tuple` literal (rewrite as a `dict` literal)
   67	C408   	Unnecessary `dict` call (rewrite as a literal)
    3	C416   	Unnecessary `list` comprehension (rewrite using `list()`)
   57	TID252 	Relative imports from parent modules are banned
    3	RET501 	Do not explicitly `return None` in function if it is the only possible return value
   26	RET502 	Do not implicitly `return None` in function able to return non-`None` value
   83	RET503 	Missing explicit `return` at the end of function able to return non-`None` value
   91	RET504 	Unnecessary variable assignment before `return` statement
  231	RET505 	Unnecessary `else` after `return` statement
   25	RET506 	Unnecessary `else` after `raise` statement
    3	RET507 	Unnecessary `elif` after `continue` statement
    1	RET508 	Unnecessary `elif` after `break` statement
    3	ISC001 	Implicitly concatenated string literals on one line
   23	ISC003 	Explicitly concatenated string should be implicitly concatenated
  193	T201   	`print` found
17654	Q000   	Single quotes found but double quotes preferred
    8	Q001   	Single quote multiline found but double quotes preferred
    2	Q002   	Single quote docstring found but double quotes preferred
 3581	ANN001 	Missing type annotation for function argument `name`
   59	ANN002 	Missing type annotation for `*args`
  145	ANN003 	Missing type annotation for `**kargs`
 2017	ANN101 	Missing type annotation for `self` in method
  132	ANN102 	Missing type annotation for `cls` in classmethod
 2817	ANN201 	Missing return type annotation for public function `get_group`
  382	ANN202 	Missing return type annotation for private function `query`
  201	ANN204 	Missing return type annotation for special method `__init__`
   19	ANN205 	Missing return type annotation for staticmethod `find`
   93	ANN206 	Missing return type annotation for classmethod `generate_random_password`
    7	ANN401 	Dynamically typed expressions (typing.Any) are disallowed in `get_users_logged_books`
   73	SIM115 	Use context handler for opening files
    1	SIM101 	Multiple `isinstance` calls for `val`, merge into a single call
   25	SIM102 	Use a single `if` statement instead of nested `if` statements
    1	SIM103 	Return the condition `re.search(params["pattern"], web.ctx.path)` directly
   10	SIM105 	Use `contextlib.suppress(UniqueViolation, IntegrityError)` instead of try-except-pass
   68	SIM108 	Use ternary operator `what = "count(distinct(thing_id)) as count" if distinct else "count(thing_id) as count"` instead of if-else-block
    1	SIM110 	Use `return any(web.ctx.path.startswith(path_segment) for path_segment in self.cors_prefixes)` instead of `for` loop
    4	SIM111 	Use `return all(substr_match(i, j) for i, j in zip(pub1, pub2))` instead of `for` loop
    1	SIM117 	Use a single `with` statement with multiple contexts instead of nested `with` statements
    1	SIM118 	Use `key in language.name_translated` instead of `key in language.name_translated.keys()`
    1	SIM201 	Use `i.sort_by_count != "false"` instead of `not i.sort_by_count == "false"`
    1	SIM210 	Use `bool(i.dry_run)` instead of `True if i.dry_run else False`
   15	SIM300 	Yoda conditions are discouraged, use `root.tag == '{http://www.w3.org/1999/02/22-rdf-syntax-ns#}RDF'` instead
    1	SIM401 	Use `result = result.get(req, [])` instead of an `if` block
  146	UP007  	Use `X | Y` for type annotations
    2	UP031  	Use format specifiers instead of percent format
    7	UP035  	Import from `collections.abc` instead: `Callable`
  182	D100   	Missing docstring in public module
  505	D101   	Missing docstring in public class
 1542	D102   	Missing docstring in public method
  881	D103   	Missing docstring in public function
   35	D104   	Missing docstring in public package
   46	D105   	Missing docstring in magic method
    1	D106   	Missing docstring in public nested class
  117	D107   	Missing docstring in `__init__`
  133	D200   	One-line docstring should fit on one line
   40	D202   	No blank lines allowed after function docstring (found 1)
  373	D205   	1 blank line required between summary line and description
    2	D208   	Docstring is over-indented
   21	D209   	Multi-line docstring closing quotes should be on a separate line
    3	D210   	No whitespaces allowed surrounding docstring text
    2	D211   	No blank lines allowed before class docstring
  392	D212   	Multi-line docstring summary should start at the first line
   41	D300   	Use """triple double quotes"""
    5	D301   	Use r""" if any backslashes in a docstring
  320	D400   	First line should end with a period
  464	D401   	First line of docstring should be in imperative mood: "Returns the group named 'name'."
   24	D402   	First line should not be the function's signature
    8	D403   	First word of the first line should be properly capitalized
   14	D404   	First word of the docstring should not be "This"
    4	D406   	Section name should end with a newline ("Returns")
    8	D407   	Missing dashed underline after section ("Args")
    1	D411   	Missing blank line before section ("Returns")
  314	D415   	First line should end with a period, question mark, or exclamation point
    2	D417   	Missing argument descriptions in the docstring: `displayname`, `email`, `password`, `retries`, `test`, `username`, `verified`
    2	D419   	Docstring is empty
  293	N801   	Class name `home` should use CapWords convention
  336	N802   	Function name `GET` should be lowercase
    7	N805   	First argument of a method should be named `self`
   38	N806   	Variable `STRIP_CHARS` in function should be lowercase
    4	N812   	Lowercase `etree` imported as non-lowercase `ET`
    3	N813   	Camelcase `IOBase` imported as lowercase `file`
    6	N815   	Variable `opdsNS` in class scope should not be mixedCase
    1	N816   	Variable `errorLookup` in global scope should not be mixedCase
    1	N817   	Camelcase `ElementTree` imported as acronym `ET`
   14	N818   	Exception name `InvalidType` should be named with an Error suffix
  267	I001   	Import block is un-sorted or un-formatted
  185	ERA001 	Found commented-out code
 1409	S101   	Use of `assert` detected
    1	S102   	Use of `exec` detected
    3	S103   	`os.chmod` setting a permissive mask `0o666` on file or directory
    3	S104   	Possible binding to all interfaces
    7	S105   	Possible hardcoded password: "foobar"
    8	S106   	Possible hardcoded password: "openlibrary"
    7	S108   	Probable insecure usage of temporary file or directory: "/tmp/"
    6	S110   	`try`-`except`-`pass` detected, consider logging the exception
   54	S113   	Probable use of requests call without timeout
    6	S324   	Probable use of insecure hash functions in `hashlib`: "md5"
    1	S506   	Probable use of unsafe `yaml.load`. Allows instantiation of arbitrary objects. Consider `yaml.safe_load`.
   39	FBT001 	Boolean positional arg in function definition
  156	FBT002 	Boolean default value in function definition
    9	FBT003 	Boolean positional value in function call
  128	ARG001 	Unused function argument: `kargs`
  142	ARG002 	Unused method argument: `ocaid`
    5	ARG003 	Unused class method argument: `test`
   41	ARG005 	Unused lambda argument: `olid`
   26	DTZ001 	The use of `datetime.datetime()` without `tzinfo` argument is not allowed
    5	DTZ002 	The use of `datetime.datetime.today()` is not allowed, use `datetime.datetime.now(tz=)` instead
   54	DTZ003 	The use of `datetime.datetime.utcnow()` is not allowed, use `datetime.datetime.now(tz=)` instead
    4	DTZ004 	The use of `datetime.datetime.utcfromtimestamp()` is not allowed, use `datetime.datetime.fromtimestamp(ts, tz=)` instead
   47	DTZ005 	The use of `datetime.datetime.now()` without `tz` argument is not allowed
    2	DTZ006 	The use of `datetime.datetime.fromtimestamp()` without `tz` argument is not allowed
   10	DTZ007 	The use of `datetime.datetime.strptime()` without %z must be followed by `.replace(tzinfo=)` or `.astimezone()`
   15	DTZ011 	The use of `datetime.date.today()` is not allowed, use `datetime.datetime.now(tz=).date()` instead
    2	PGH001 	No builtin `eval()` allowed
    4	PGH003 	Use specific rule codes when ignoring type issues
    4	PGH004 	Use specific rule codes when using `noqa`
   91	EM101  	Exception must not use a string literal, assign to variable first
   22	EM102  	Exception must not use an f-string literal, assign to variable first
   26	PT001  	Use `@pytest.fixture()` over `@pytest.fixture`
    9	PT004  	Fixture `add_languages` does not return anything, add leading underscore
   31	PT006  	Wrong name(s) type in `@pytest.mark.parametrize`, expected `tuple`
    1	PT007  	Wrong values type in `@pytest.mark.parametrize` expected `list` of `tuple`
   49	PT009  	Use a regular `assert` instead of unittest-style `assertEqual`
    1	PT012  	`pytest.raises()` block should contain a single simple statement
    1	PT015  	Assertion always fails, replace with `pytest.fail()`
    1	PT017  	Found assertion on exception `e` in except block, use `pytest.raises()` instead
    9	PT018  	Assertion should be broken down into multiple parts
    5	PT021  	Use `yield` instead of `request.addfinalizer`
   29	PT023  	Use `@pytest.mark.xfail()` over `@pytest.mark.xfail`
    7	PIE790 	Unnecessary `pass` statement
    2	PIE804 	Unnecessary `dict` kwargs
    1	PIE807 	Prefer `list` over useless lambda
  970	COM812 	Trailing comma missing
   14	INP001 	File `openlibrary/catalog/marc/tests/test_get_subjects.py` is part of an implicit namespace package. Add an `__init__.py`.
    9	EXE001 	Shebang is present but file is not executable
    1	EXE002 	The file is executable but no shebang is present
   11	TCH001 	Move application import `openlibrary.core.lists.model.ListMixin` into a type-checking block
    4	TCH002 	Move third-party import `babel.messages.catalog.Message` into a type-checking block
   25	TCH003 	Move standard library import `collections.abc.Iterator` into a type-checking block
   23	TRY002 	Create your own exception
   77	TRY003 	Avoid specifying long messages outside the exception class
    3	TRY004 	Prefer `TypeError` exception for invalid type
   31	TRY200 	Use `raise from` to specify exception cause
    5	TRY201 	Use `raise` without specifying exception name
   37	TRY300 	Consider moving this statement to an `else` block
    9	TRY301 	Abstract `raise` to an inner function
   33	TRY400 	Use `logging.exception` instead of `logging.error`
    6	PTH100 	`os.path.abspath` should be replaced by `.resolve()`
    3	PTH101 	`os.chmod` should be replaced by `.chmod()`
    1	PTH102 	`os.mkdir` should be replaced by `.mkdir()`
   12	PTH103 	`os.makedirs` should be replaced by `.mkdir(parents=True)`
    5	PTH107 	`os.remove` should be replaced by `.unlink()`
    2	PTH108 	`os.unlink` should be replaced by `.unlink()`
    1	PTH109 	`os.getcwd` should be replaced by `Path.cwd()`
   35	PTH110 	`os.path.exists` should be replaced by `.exists()`
    3	PTH111 	`os.path.expanduser` should be replaced by `.expanduser()`
    3	PTH112 	`os.path.isdir` should be replaced by `.is_dir()`
    3	PTH116 	`os.stat` should be replaced by `.stat()` or `.owner()` or `.group()`
   61	PTH118 	`os.path.join` should be replaced by foo_path / "bar"
    1	PTH119 	`os.path.basename` should be replaced by `.name`
   34	PTH120 	`os.path.dirname` should be replaced by `.parent`
    2	PTH122 	`os.path.splitext` should be replaced by `.suffix`
  122	PTH123 	`open("foo")` should be replaced by `Path("foo").open()`
   15	G002   	Logging statement uses `%`
    3	G003   	Logging statement uses `+`
   66	G004   	Logging statement uses f-string
    8	G010   	Logging statement uses `warn` instead of `warning`
   13	G201   	Logging `.exception(...)` should be used instead of `.error(..., exc_info=True)`
   64	RSE102 	Unnecessary parentheses on raised exception
  105	SLF001 	Private member accessed: `infogami._setup`
   17	RUF001 	String contains ambiguous unicode character 'α' (did you mean 'a'?)
    4	RUF002 	Docstring contains ambiguous unicode character 'В' (did you mean 'B'?)
   11	RUF005 	Consider `[sys.argv[0], *list(args)]` instead of concatenation
    6	RUF100 	Unused blanket `noqa` directive
```

Resolving patch:
diff --git a/openlibrary/catalog/marc/marc_subject.py b/openlibrary/catalog/marc/marc_subject.py
--- a/openlibrary/catalog/marc/marc_subject.py
+++ b/openlibrary/catalog/marc/marc_subject.py
@@ -2,8 +2,8 @@
     openlibrary.catalog.marc.get_subjects is the preferred module
 """
 
-# Tell the flake8 linter to ignore this deprecated file.
-# flake8: noqa
+# Tell the ruff linter to ignore this deprecated file.
+# ruff: noqa
 
 from collections import defaultdict
 from deprecated import deprecated


Test patch:
diff --git a/requirements_test.txt b/requirements_test.txt
--- a/requirements_test.txt
+++ b/requirements_test.txt
@@ -4,9 +4,9 @@
 
 -r requirements.txt
 debugpy>=1.6.4
-flake8==6.0.0
-mypy==1.0.0
+mypy==1.1.1
 pymemcache==4.0.0
-pytest==7.2.1
+pytest==7.2.2
 pytest-asyncio==0.20.3
+ruff==0.0.256
 safety==2.3.5
diff --git a/scripts/test_py3.sh b/scripts/test_py3.sh
--- a/scripts/test_py3.sh
+++ b/scripts/test_py3.sh
@@ -7,7 +7,7 @@ pytest . \
        --ignore=vendor
 RETURN_CODE=$?
 
-flake8 --exit-zero --count --select=E722,F403 --show-source --statistics  # Show bare exceptions and wildcard (*) imports
+ruff --exit-zero --select=E722,F403 --show-source  # Show bare exceptions and wildcard (*) imports
 safety check || true  # Show any insecure dependencies
 
 exit ${RETURN_CODE}