Repository: python/mypy

Issue:
[Stubgen bug] Fix behavior for Instance Variables in C extensions
Stubgen generates wrong stubs for instance variables defined in C extensions.

In the attached `extension.C`, I've created a class named _Custom_ which has 3 instance variables: _var1_,_var2_,_var3_. On running stubgen for this extension, the generated .pyi file contains: 

```
from typing import Any, ClassVar

class Custom:
    // This is wrong. Var{1,2,3} are not Class Variables but are rather instance variables.
    var1: ClassVar[getset_descriptor] = ...
    var2: ClassVar[getset_descriptor] = ...
    var3: ClassVar[member_descriptor] = ...   
    // This is wrong. 

    def display(self, *args, **kwargs) -> Any: ...
```

**Detailed Reproducer:**
 1. In some test directory (say $DIR), download the attached file _extension.zip_ to extension folder.
 3. Compile the extension by running: `g++ -fPIC -std=c++11 -I. -I/opt/python/python-3.9/include/python3.9 -Iextension -c extension/extension.C -o extension.o` in directory $DIR.
 4. Create the .so file by running: `g++ -shared  -L/opt/python/python-3.9/lib64 -Wl,-R/opt/python/python-3.9/lib64 extension.o -o extension.so` in directory $DIR.
 5. Run stubgen using `env PYTHONPATH=$DIR stubgen -m extension`
 6. Notice that the generated $DIR/out/extension.pyi declares var{1,2,3} as 'Class Variables' which is wrong.

On digging into this, I found this is due to a wrong check in `stubgenc.py#is_c_property`.
```
def is_c_property(obj: object) -> bool:
    return inspect.isdatadescriptor(obj) and hasattr(obj, 'fget')
```
The check should contain "or" (instead of "and") since the instance variables in case of C extension doesn't have the "fget" attribute but `inspect.isdatadescription` return True.
On making this change locally, stubgen runs fine. 

If this looks good to you, I can raise a PR for the same.

[extension.zip](https://github.com/python/mypy/files/8023192/extension.zip)
[Stubgen bug] Fix behavior for Instance Variables in C extensions
Stubgen generates wrong stubs for instance variables defined in C extensions.

In the attached `extension.C`, I've created a class named _Custom_ which has 3 instance variables: _var1_,_var2_,_var3_. On running stubgen for this extension, the generated .pyi file contains: 

```
from typing import Any, ClassVar

class Custom:
    // This is wrong. Var{1,2,3} are not Class Variables but are rather instance variables.
    var1: ClassVar[getset_descriptor] = ...
    var2: ClassVar[getset_descriptor] = ...
    var3: ClassVar[member_descriptor] = ...   
    // This is wrong. 

    def display(self, *args, **kwargs) -> Any: ...
```

**Detailed Reproducer:**
 1. In some test directory (say $DIR), download the attached file _extension.zip_ to extension folder.
 3. Compile the extension by running: `g++ -fPIC -std=c++11 -I. -I/opt/python/python-3.9/include/python3.9 -Iextension -c extension/extension.C -o extension.o` in directory $DIR.
 4. Create the .so file by running: `g++ -shared  -L/opt/python/python-3.9/lib64 -Wl,-R/opt/python/python-3.9/lib64 extension.o -o extension.so` in directory $DIR.
 5. Run stubgen using `env PYTHONPATH=$DIR stubgen -m extension`
 6. Notice that the generated $DIR/out/extension.pyi declares var{1,2,3} as 'Class Variables' which is wrong.

On digging into this, I found this is due to a wrong check in `stubgenc.py#is_c_property`.
```
def is_c_property(obj: object) -> bool:
    return inspect.isdatadescriptor(obj) and hasattr(obj, 'fget')
```
The check should contain "or" (instead of "and") since the instance variables in case of C extension doesn't have the "fget" attribute but `inspect.isdatadescription` return True.
On making this change locally, stubgen runs fine. 

If this looks good to you, I can raise a PR for the same.

[extension.zip](https://github.com/python/mypy/files/8023192/extension.zip)


Discussion:
Hi, did anyone get a chance to look at the reproducer?
Happy to add more details if required.
Gentle ping for the maintainers to have a look at this.
Mypy is a volunteer-run project with limited resources. Any fix for this issue will likely have to come from an external contributor.
Hi, did anyone get a chance to look at the reproducer?
Happy to add more details if required.
Gentle ping for the maintainers to have a look at this.
Mypy is a volunteer-run project with limited resources. Any fix for this issue will likely have to come from an external contributor.

Resolving patch:
diff --git a/mypy/stubgenc.py b/mypy/stubgenc.py
--- a/mypy/stubgenc.py
+++ b/mypy/stubgenc.py
@@ -121,11 +121,11 @@ def is_c_classmethod(obj: object) -> bool:
 
 
 def is_c_property(obj: object) -> bool:
-    return inspect.isdatadescriptor(obj) and hasattr(obj, 'fget')
+    return inspect.isdatadescriptor(obj) or hasattr(obj, 'fget')
 
 
 def is_c_property_readonly(prop: Any) -> bool:
-    return prop.fset is None
+    return hasattr(prop, 'fset') and prop.fset is None
 
 
 def is_c_type(obj: object) -> bool:
@@ -287,6 +287,10 @@ def infer_prop_type(docstr: Optional[str]) -> Optional[str]:
         else:
             return None
 
+    # Ignore special properties/attributes.
+    if name.startswith('__') and name.endswith('__'):
+        return
+
     inferred = infer_prop_type(getattr(obj, '__doc__', None))
     if not inferred:
         fget = getattr(obj, 'fget', None)


Test patch:
diff --git a/mypy/test/teststubgen.py b/mypy/test/teststubgen.py
--- a/mypy/test/teststubgen.py
+++ b/mypy/test/teststubgen.py
@@ -20,7 +20,8 @@
 )
 from mypy.stubutil import walk_packages, remove_misplaced_type_comments, common_dir_prefix
 from mypy.stubgenc import (
-    generate_c_type_stub, infer_method_sig, generate_c_function_stub, generate_c_property_stub
+    generate_c_type_stub, infer_method_sig, generate_c_function_stub, generate_c_property_stub,
+    is_c_property_readonly
 )
 from mypy.stubdoc import (
     parse_signature, parse_all_signatures, build_signature, find_unique_signatures,
@@ -868,9 +869,34 @@ def get_attribute(self) -> None:
                 pass
             attribute = property(get_attribute, doc="")
 
-        output: List[str] = []
-        generate_c_property_stub('attribute', TestClass.attribute, [], [], output, readonly=True)
-        assert_equal(output, ['@property', 'def attribute(self) -> str: ...'])
+        readwrite_properties: List[str] = []
+        readonly_properties: List[str] = []
+        generate_c_property_stub('attribute', TestClass.attribute, [],
+                                 readwrite_properties, readonly_properties,
+                                 is_c_property_readonly(TestClass.attribute))
+        assert_equal(readwrite_properties, [])
+        assert_equal(readonly_properties, ['@property', 'def attribute(self) -> str: ...'])
+
+    def test_generate_c_property_with_rw_property(self) -> None:
+        class TestClass:
+            def __init__(self) -> None:
+                self._attribute = 0
+
+            @property
+            def attribute(self) -> int:
+                return self._attribute
+
+            @attribute.setter
+            def attribute(self, value: int) -> None:
+                self._attribute = value
+
+        readwrite_properties: List[str] = []
+        readonly_properties: List[str] = []
+        generate_c_property_stub("attribute", type(TestClass.attribute), [],
+                                 readwrite_properties, readonly_properties,
+                                 is_c_property_readonly(TestClass.attribute))
+        assert_equal(readwrite_properties, ['attribute: Any'])
+        assert_equal(readonly_properties, [])
 
     def test_generate_c_type_with_single_arg_generic(self) -> None:
         class TestClass: