Repository: praw-dev/praw

Issue:
dir changes attributes of praw.objects.Subreddits in place
Today I noticed that the built-in function [`dir()`](https://docs.python.org/2/library/functions.html#dir) seems to change the attributes of praw.objects.Subreddits in place.

![screen shot 2016-03-09 at 4 01 22 pm](https://cloud.githubusercontent.com/assets/12176060/13650376/69edcf1e-e610-11e5-9e7d-5ef2051ad2d7.png)

Could this possibly be the result of a buggy `__dir__()` implementation in a praw.objects.Subreddit parent class? I honestly have no clue what to make of this.



Discussion:
It's possible that something changes as a result, e.g., the lazy submission object no longer becomes lazy. It'd be helpful if you could pinpoint what attributes are changing, and explain why having stability across dir is useful. Then someone, or you, could look into resolving those issues.

So I checked and it seems like the first call to `dir()` is missing these 41 Subreddit member variables.

```
accounts_active             banner_img                  banner_size                 
collapse_deleted_comments   comment_score_hide_mins     created                     
created_utc                 description                 description_html            
header_img                  header_size                 header_title                
hide_ads                    icon_img                    icon_size                   
id                          key_color                   lang                        
name                        over18                      public_description          
public_description_html     public_traffic              quarantine                  
submission_type             submit_link_label           submit_text                 
submit_text_html            submit_text_label           subreddit_type              
subscribers                 suggested_comment_sort      title                       
url                         user_is_banned              user_is_contributor         
user_is_moderator           user_is_muted               user_is_subscriber          
user_sr_theme_enabled       wiki_enabled   
```

and these 10 Redditor member variables

```
comment_karma               created                     created_utc                 
has_verified_email          hide_from_robots            id                          
is_friend                   is_gold                     is_mod                      
link_karma    
```

so it seems like all of the missing attributes are variables and not methods.

In general, you could imagine the fact that

``` python
prog = r.get_subreddit('programming')
print 'description' in dir(prog) and hasattr(prog, 'description')
```

prints `False` might trip some people up.  For example, if someone is writing a function that checks for the existence of a class method, they would get different answers for `'description' in dir(prog)` and `hasattr(prog, 'description')`, which could lead to a hard to find bug.

I definitely agree that `dir()` stability is not the highest priority, so I wouldn't mind looking into it.

Because of the lazy loading many of the attributes aren't available until they are accessed the first time. That is by design. I suppose `hasattr(prog, 'description')` would have the side-effect of actually fetching the needed data. However, it seems odd that `dir` is doing that. The latter may be worth investigating.

The issue doesn't reproduce exactly for me:

![image](https://cloud.githubusercontent.com/assets/7299570/13685715/aec7d524-e6c6-11e5-9456-bbb6730a130c.png)

It's clear that when I deepcopy the object, one of the attribute copies triggers the lazy load, and both objects have the full API info. However, the simple `dir` test didn't have the effect that you showed in your screenshot, neither of the objects has the extended list.

Could this have anything to do with the environment you're using to run Python? Do you get the same effect using a normal terminal or a script? I'm just throwing out ideas.

Hmmm... interesting.  This is what I get running those commands

<img width="527" alt="screen shot 2016-03-10 at 7 28 13 pm" src="https://cloud.githubusercontent.com/assets/12176060/13689369/651e08b4-e6f6-11e5-973c-153b662a389b.png">

It looks like bboe is right... the output is actually "correct," it's just not clear why `dir()` fetches new data under my configuration. Probably something to do with 2.7 vs 3.5.

Okay, must be the difference between Python 2 and 3, that's an interesting distinction. Does `two.__dict__` instead of `dir(two)` cause the same thing to happen?

If this is something that can be fixed in a clean and version-compatible way, I'd say there's no reason not to, but I'm not sure how you could change the behavior of something as basic as `dir`. It appears (in 3.5), that it can be overridden:

```
>>> class A:
...     def __dir__(self):
...             return [1, 2, 3]
...
>>>
>>> a=A()
>>> dir(a)
[1, 2, 3]
```

I don't know what you would put inside that method that achieves all the normal functionality of `dir` and doesn't do whatever causes Python 2 to trip the lazy loader. Plus it just seems like something that should never have to be done. You mentioned

> if someone is writing a function that checks for the existence of a class method, they would get different answers for 'description' in dir(prog)

but I think `hasattr` is always the correct answer anyway. We should have a very good reason before overriding this kind of thing.

``` python
class Test(object):
    def __init__(self):
        self.attribute = True

    def __getattr__(self, attr):
        print('Fetching {}'.format(attr))
        return super(Test, self).__getattr__(attr)


test = Test()
print(dir(test) == dir(test))
```

``` sh
python2 ./tmp.py 
Fetching __members__
Fetching __methods__
Fetching __members__
Fetching __methods__
True
```

```
python3 ./tmp.py 
True
```

@bboe Nice demonstration of what Python2 is doing differently from Python3. Since the `__getattr__` method of RedditContentObjects performs the API call, maybe we can add some lines that prevent the fetch if the requested attribute is either `__members__` or `__methods__`? We already have `__setstate__` being [blacklisted](https://github.com/praw-dev/praw/blob/b3db45c931cd1371a932b60df025bd5c82ab639d/praw/objects.py#L83) so it isn't too much of a stretch.

I notice that your example didn't print `Fetching attribute` so it looks like the other properties of the object will not have any negative effects.

Yes, that would be the solution. Would you like to push that through @voussoir?

Sure thing! Does Python expect to receive something when getting those two attributes, or should they return None / raise AttributeError? I'm trying Python2 on [repl.it](https://repl.it/) but your example is actually giving `AttributeError: 'super' object has no attribute '__getattr__'`, so I'm not quite sure what those two attributes are meant to be for.

> return super(Test, self).**getattr**(attr) 

That's not actually the right thing to do. I think we only need to add exceptions for those two methods exactly as you described (i.e., similarly to `__setstate__`).

Okay, I didn't mean I would add the `super` part, I was just playing with your example because I wanted to see if Python 2 expected to receive anything from those values. It looks like they aren't necessary, so they'll just raise AttributeError.

I'll have a PR open in just a few minutes!


Resolving patch:
diff --git a/praw/objects.py b/praw/objects.py
--- a/praw/objects.py
+++ b/praw/objects.py
@@ -80,7 +80,12 @@ def __hash__(self):
 
     def __getattr__(self, attr):
         """Return the value of the `attr` attribute."""
-        if attr != '__setstate__' and not self._has_fetched:
+        # Because this method may perform web requests, there are certain
+        # attributes we must blacklist to prevent accidental requests:
+        # __members__, __methods__: Caused by `dir(obj)` in Python 2.
+        # __setstate__: Caused by Pickle deserialization.
+        blacklist = ('__members__', '__methods__', '__setstate__')
+        if attr not in blacklist and not self._has_fetched:
             self._has_fetched = self._populate(None, True)
             return getattr(self, attr)
         msg = '\'{0}\' has no attribute \'{1}\''.format(type(self), attr)


Test patch: