Repository: cython/cython
Pull request: Describe refcount behaviour of object vs PyObject*
Fixes #2589

Problem statement:
`object` vs `PyObject*`
From my (limited) understanding from reading the Cython includes directory along with the CPython API, in many cases it seems that `object` is used interchangeably with `PyObject*`. However, the compiler complains about not being able to convert between the two when again, they appear to be interchangeable.

So it seems that one of the following holds true:
- `object` and `PyObject*` are equivalent and the Cython compiler has a bug where it does not consider `object` a shorthand for `PyObject*`.
- `object` and `PyObject*` are not equivalent and the `Includes` directory has `.pxd` files with wrong specifications.

Hints from issue discussion:
Hmm, right, that's probably worth explaining somewhere.

`object` is a refcounted owned reference, `PyObject*` is a non-refcounted (usually borrowed) pointer to an object. There is no dedicated borrowed reference type in Cython, that's why we use the pointer type for it. Converting between a pointer (any pointer in fact, also `void*`) and an owned object reference is done through a cast, either `obj = <object>ptr` (which increases the refcount => owned reference) or `ptr = <PyObject*>obj` (which leaves the refcount allone => borrowed reference).

PR welcome that puts that explanation into a reasonable place.
One place where this could be clarified here: https://cython.readthedocs.io/en/latest/src/userguide/language_basics.html?highlight=%22except%20%20%22#python-objects-as-parameters-and-return-values

At the C/C++ level, the function would be declared as returning `PyObject*`, but in Cython it can be declared as returning `object` explicitly as well as implicitly.

Commit message: docs: Describe refcount behaviour of object vs PyObject*

Patch to review:
diff --git a/docs/examples/userguide/language_basics/parameter_refcount.pyx b/docs/examples/userguide/language_basics/parameter_refcount.pyx
new file mode 100644
index 00000000000..c9b14b06425
--- /dev/null
+++ b/docs/examples/userguide/language_basics/parameter_refcount.pyx
@@ -0,0 +1,20 @@
+from __future__ import print_function
+
+from cpython.ref cimport PyObject
+
+import sys
+
+python_string = "foo"
+python_string_refcount = sys.getrefcount(python_string)
+
+cdef owned_reference(object obj):
+    refcount = sys.getrefcount(python_string)
+    print('Inside owned_reference: {refcount}'.format(refcount=refcount))
+
+cdef borrowed_reference(PyObject * obj):
+    refcount = obj.ob_refcnt
+    print('Inside borrowed_reference: {refcount}'.format(refcount=refcount))
+
+print('Initial refcount: {refcount}'.format(refcount=python_string_refcount))
+owned_reference(python_string)
+borrowed_reference(<PyObject *>python_string)
diff --git a/docs/src/userguide/language_basics.rst b/docs/src/userguide/language_basics.rst
index b5746e318fb..08515b712ed 100644
--- a/docs/src/userguide/language_basics.rst
+++ b/docs/src/userguide/language_basics.rst
@@ -288,6 +288,17 @@ In the interests of clarity, it is probably a good idea to always be explicit
 about object parameters in C functions.
 
 
+To create a borrowed reference, specify the parameter type as ``PyObject*``.
+Cython won't perform automatic ``Py_INCREF``, or ``Py_DECREF``, e.g.:
+
+.. literalinclude:: ../../examples/userguide/language_basics/parameter_refcount.pxd
+
+will display::
+
+    Initial refcount: 3
+    Inside owned_reference: 4
+    Inside borrowed_reference: 3
+
 .. _optional_arguments:
 
 Optional Arguments
@@ -579,6 +590,10 @@ Here is an example:
 
 The precedence of ``<...>`` is such that ``<type>a.b.c`` is interpreted as ``<type>(a.b.c)``.
 
+Casting to ``<object>`` creates an owned reference. Cython will automatically
+perform a ``Py_INCREF`` and ``Py_DECREF`` operation. Casting to
+``<PyObject *>`` creates a borrowed reference, leaving the refcount unchanged.
+
 .. _checked_type_casts:
 
 Checked Type Casts


Code review comments:
diff_hunk: @@ -0,0 +1,20 @@
+from __future__ import print_function
+
+from cpython.ref cimport PyObject
+
+import sys
+
+python_string = "foo"
+python_string_refcount = sys.getrefcount(python_string)
path: docs/examples/userguide/language_basics/parameter_refcount.pyx
text: @user1:
This is a bit risky in a test. Python string literals are interned by Cython, which means that their exact refcount might depend on other modules, and therefore not be predictable. You can avoid that by constructing a new string object with some non-optimisable operation, e.g. `string.ascii_letters * 2`.

Or be less explicit about the exact refcounts in the corresponding documentation than saying "will display". :)

@author:
I'd like to keep the wording as is. I've changed the example to use a dict, in case CPython 3.9+ gets smarter about optimizing constant str initializations.