Repository: cython/cython

Issue:
Document compilation caching?
### Describe your issue

I notice that Cython seems to have a ccache-like feature that is disabled by default and that does not seem documented. 
https://github.com/cython/cython/blob/f3cad15100da00936b3ac31865c28d22eb95c108/Cython/Compiler/Options.py#L671-L672

Unless there is a definite reason for not expose it to end users (though it does seem [tested](https://github.com/cython/cython/blob/master/Cython/Build/Tests/TestCyCache.py)), it would be nice to document it.


Discussion:
cc @jorisvandenbossche
Hmm, right. That seems worth documenting. I can imagine that it can make a real difference for large projects, as it apparently did for Sagemath at the time (for which it seems to have been implemented).

There are two options: `cache` allows enabling the cache by setting it to either `True` or the path of a cache directory, and `cache_size` allows to change the maintained size of the LRU cache.
By the way, I'm not sure it's related, but I see a subdirectory `.cython/inline` on my machine, and the only file it contains seems to have been generated by 0.28a0. Is it failed experiment?
Ok, more annoyingly, it doesn't seem possible to pass the `cache` option on the `python -m cython` command line, unlike `cythonize` which seems to have a `-s` option for that.

I'm not sure I understand the difference between "compilation directives" and "compilation options", but apparently `python -m cython` only accepts "directives" not "options".

(why do we have both `python -m cython` and `cythonize`?)
And if I add `# cython: cache = True` inside a source file, it seems ignored (nothing appears in `.cython`). Which is understandable, but it means that for now it is not possible to enable caching in PyArrow.
`cythonize` is a "do it all in one go" kind of tool, whereas `cython` (or `python -m cython`) is a "translate a single file and let me do the rest" tool. Both have their use cases.

And yes, it seems that `cache` is not available from the command line of the `cython` tool. It's a compile option, not a directive that you can set from a source file.

Unknown directives should get rejected, though, and not pass silently.
It _is_ rejected:
```
  [1/111] cd /tmp/tmpt22p8ntp.build-temp && /home/antoine/mambaforge/envs/pyarrow/bin/python3.10 -m cython --cplus --gdb --warning-errors --no-c-in-traceback -Xcache=False --directive embedsignature=True --working /home/antoine/arrow/dev/python --output-file /tmp/tmpt22p8ntp.build-temp/_dataset.cpp /home/antoine/arrow/dev/python/pyarrow/_dataset.pyx
  FAILED: CMakeFiles/_dataset_pyx /tmp/tmpt22p8ntp.build-temp/CMakeFiles/_dataset_pyx
  cd /tmp/tmpt22p8ntp.build-temp && /home/antoine/mambaforge/envs/pyarrow/bin/python3.10 -m cython --cplus --gdb --warning-errors --no-c-in-traceback -Xcache=False --directive embedsignature=True --working /home/antoine/arrow/dev/python --output-file /tmp/tmpt22p8ntp.build-temp/_dataset.cpp /home/antoine/arrow/dev/python/pyarrow/_dataset.pyx
  Traceback (most recent call last):
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/site-packages/cython.py", line 17, in <module>
      main(command_line = 1)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 762, in main
      options, sources = parse_command_line(args)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/site-packages/Cython/Compiler/CmdLine.py", line 217, in parse_command_line
      arguments, sources = parse_command_line_raw(parser, args)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/site-packages/Cython/Compiler/CmdLine.py", line 192, in parse_command_line_raw
      arguments, unknown = parser.parse_known_args(args_without_embed)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/argparse.py", line 1866, in parse_known_args
      namespace, args = self._parse_known_args(args, namespace)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/argparse.py", line 2079, in _parse_known_args
      start_index = consume_optional(start_index)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/argparse.py", line 2019, in consume_optional
      take_action(action, args, option_string)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/argparse.py", line 1943, in take_action
      action(self, namespace, argument_values, option_string)
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/site-packages/Cython/Compiler/CmdLine.py", line 22, in __call__
      directives = Options.parse_directive_list(
    File "/home/antoine/mambaforge/envs/pyarrow/lib/python3.10/site-packages/Cython/Compiler/Options.py", line 533, in parse_directive_list
      raise ValueError('Unknown option: "%s"' % name)
  ValueError: Unknown option: "cache"
```

It would be nice if some way of enabling cache from the `cython` command (or a global config file) was available. There does not seem to be any functional reason for caching to be usable from `cythonize` but not `cython`.
> And yes, it seems that cache is not available from the command line of the cython tool. It's a compile option, not a directive that you can set from a source file.

Yes but it is more complicated. Cache is hardwired into cythonize code so it is not just adding new option to `cython`. We need to extract cache code to separate module and integrate it in both `cython` and `cythonize` or more deeper in common compilation code. Cannot tell right now what is better out of blue.
> Cache is hardwired into cythonize code so it is not just adding new option to `cython`.

That makes sense, given the different nature and complexity of both tools.

> It would be nice if some way of enabling cache from the cython command (or a global config file) was available. 

I have created PR adding cache support to cython command - see #6091

Resolving patch:
diff --git a/Cython/Build/Dependencies.py b/Cython/Build/Dependencies.py
--- a/Cython/Build/Dependencies.py
+++ b/Cython/Build/Dependencies.py
@@ -997,6 +997,9 @@ def cythonize(module_list, exclude=None, nthreads=0, aliases=None, quiet=False,
                                 See :ref:`compiler-directives`.
 
     :param depfile: produce depfiles for the sources if True.
+    :param cache: If ``True`` the cache enabled with default path. If the value is a path to a directory,
+                  then the directory is used to cache generated ``.c``/``.cpp`` files. By default cache is disabled.
+                  See :ref:`cython-cache`.
     """
     if exclude is None:
         exclude = []


Test patch: