Repository: uqfoundation/dill

Issue:
Infinite recursion in pickling dynamically generated classes
Consider

```
def createB():       
     class B(object):
       def __init__(self):
         super(B, self).__init__()
     return B
```

Now `dill.dumps(createB())` fails with an infinite loop of

```
T2: <class '__main__.B'>
D2: <dict object at 0x10ef5e050>
F1: <function __init__ at 0x10f075aa0>
D1: <dict object at 0x10e7ac050>
# D1
Ce: <cell at 0x10f013e88: type object at 0x7ff6a1d7cdb0>
T2: <class '__main__.B'>
...
```


Discussion:
mmckerns: Yours is a pretty nice solution...
I think something like the following would work in python 2 (and python 3).
```
def _create_reference_cell():
    contents = [None]
    def updater(value):
        contents[0] = value
    updater(updater)
    return (lambda: contents[0]).func_closure[0]
```
anivegesana: Unfortunately, the value of the cell would become the list instead of the element inside the list, but thank you for the suggestion. I was able to use it to make a solution that works for both Python 2 and Python 3 using the `vars` function to effectively pass the scope of the outer function to an inner class. For reasons I do not understand, this doesn't work if a function is used instead of a class.

```py
def _create_reference_cell():
    contents = None
    v = vars()
    class Updater(object):
        def __call__(self, value):
            v['contents'] = value
    contents = Updater()
    return (lambda: contents).__closure__[0]
```

Also, what are you plans with regards to compatibility? In Python 3.7+, `cell_contents` is assignable directly in Python, so there is no need to coerce Python into changing the value of the variable, we can set it directly using `setattr` as in fce797e68c3c65761a9cab48fc81ada30fbd5950. Python 3.6 will reach end of life in 10 days, so we could just go the easy route and use `setattr` and not introduce any new functions to `dill._dill`. Your call.
mmckerns: I figured that either a list or a dict could serve as a stand-in for `nonlocal`.  Thanks for making the extension to python 2.7.  With regard to compatibility, we will sunset 2.7 and 3.6 soon.  The plan is to cut a release this month that is 2.7 compatible, and then turn off 2.7 shortly afterward.  Turning off 2.7 has been delayed as we have been holding on to support for pypy2.7.  I believe the following release, or so, we would then sunset 3.6.
anivegesana: Thank you for the help! I'll comb through the issues and see if there are new test cases I need to add. Will let you know when I am done.
anivegesana: I changed the names of the functions to make it easier to phase out Python 2.7 and 3.6 when they eventually are no longer in use.

Also, there is a common trend in #253 and #429. Both refer to the class inside of the code itself, which will go to the globals dictionary (__main__ in the case of both issues) to fetch the class. The problem is that the class is never written to the globals dictionary when it is loaded, so it can't look it up. There are two ways to this:

1. Update the contents of __main__ with the new class
2. Create a new copy of __main__ and use that as globals
anivegesana: I have come up with a compromise between the two options: create a `collections.ChainMap` that first accesses a dictionary containing the classes created by the Unpickler and then looks in the real global dict. This seems to be a completely different issue than the one addressed here and I will address it again later. I think this PR is good to go and ready to be looked at again.
anivegesana: The current PR is not backwards compatible in the sense that pickles made in dill 0.3.5 will not be unpickled in 0.3.4. Is this a problem?
mmckerns: @anivegesana: See https://github.com/uqfoundation/dill/issues/341.  It's not a hard requirement to maintain backward compatibility, but we try to avoid breaking backward compatibility if at all possible.
anivegesana: The only issue is that there is a new function (`_setattr`) that would be called whenever `super` is used. Instead of pickling the function as a constant, I could pickle it as `(getattr, (dill._dill, '_setattr', setattr)`, but it would add some complexity to the code. I am going to try it and see if it is worth it.
anivegesana: Alright. Now classes with `super` that were pickled in 0.3.5 can be unpickled in older versions as long as Python 3.7+ is used.

I also added a class called `BuiltinShim`. You can use it to provide version specific overloadings to builtin functions in the same way as I did. For example, imagine that `_import_module` was added in this version as an overloaded version of `importlib.import_module` but we didn't want to break functionality of older versions of dill which didn't have the `_import_module` function, we could do
```py
_import_module_shim = BuiltinShim('_import_module', importlib.import_module)
```
mmckerns: > I also added a class called `BuiltinShim`.
Interesting...

I have a release that is imminent... but I'd like to get this reviewed, merged, and included before the release.  Thanks for your work on this.
anivegesana: I actually seem to have forgot to deactivate my virtual environment when testing this. I forgot I made the argument to `_create_cell` optional to support both Python 3.6 and 3.7+. I'll fix that last thing tonight and it should be good to go.

No problem! This issue was causing problems for me and I saw that it has been on the docket for so long, so I had to spend the time to get it working.
anivegesana: I must note that this will add about a hundred bytes of bloat to every `Pickler` call that uses `super` in order to be backwards compatible with (i.e. loaded with) older `dill` versions and Python 2.7/3.6, but I designed it so that the bloat can be removed at any time at the expense of removing that support. Removing that backward compatibility won't break forward compatibility, so objects pickled with this PR can still be unpickled in versions after the bloat is removed in the future. From what I understood about your design methodology for this library, I assume that is probably the most ideal case.
anivegesana: Alright. I believe that is the last commit for this PR. It is backwards compatible with older versions of Python and dill while also having flexibility to drop the backwards compatibility to improve speed. 
anivegesana: Every time I say that, there are a couple of little things left. I can probably get rid of `_memo` in `save_function` too now, but I'll need to play around with it for a bit.

What is here now is good to go though, so if you need to push the release soon, we can leave it at that and remove `_memo` later.
mmckerns: The release won't be for a few days.  It's scheduled for this coming Monday, but is often pushed back a little.  Feel free to keep working on getting the PR how you like.  I'll probably not review this as is until tomorrow or Saturday.
anivegesana: Alright. I should be done then.

The only remaining concern I have is if my code respects the settings dictionary. If you know of any other pitfalls, please let me know.
mmckerns: The relevant part of the settings dict is (1) byref == pickle classes by reference, so this should be handled already, I think; (2) recurse == modify how globals are handled when referred to by functions, etc.  So... cursorily, I think it shouldn't have an impact on your code.
anivegesana: Sounds good. I just found that functions can also have recursive cell values, so I need to add that case to the save_function, so this might become a little more relevant for that function.

```pycon
>>> def f():
...   def g():
...     return g
...   return g
... 
>>> F=f()
>>> F.__closure__
(<cell at 0x1188a6b88: function object at 0x1192732f0>,)
>>> F
<function f.<locals>.g at 0x1192732f0>
```
mmckerns: It's unclear whether you are planning to add further changes (based on your last comment), or if you are waiting for a review of what you have now.  Would you prefer the review now?
anivegesana: Sorry. Didn't have much time to work on it. Just managed to polish it off today. There are no further changes that I plan to make and I am ready for the review.
anivegesana: Sorry about always pushing new changes when I keep saying that I am done. I figured out how to use the PyPy internal API. Now, for Python 2, _create_cell will delegate to the internal APIs of CPython and PyPy cell objects, and it will work for all versions of Python.

I promise I will not take a look at this again until you review the code and give suggestions. If I find anything new to add, it can go in the next release.

Wish you have a merry Christmas!
anivegesana: So it turns out there is no way to get it to work for PyPy 2.7. At least this pushed me to get it to work again for CPython 2.7. As promised, I won't touch this PR again except to accept the changes in your review.
mmckerns: @anivegesana: I don't mind that you keep making improvements to this PR.  Thanks for all the follow-up, and your work on this.  I'll get you a review today. Feel free to continue to make updates as needed.
anivegesana: @mmckerns I finally got a solution that works for PyPy 2.7. Unfortunately, it requires some bytecode manipulation to the `_setattr` function to create something similar to `nonlocal` in Python 2. The bytecode of the function that is being pickled is not effected, however.
mmckerns: Wow, ok... I'll take a look.  Thanks for your persistence on coverage for this issue.
mmckerns: I've worked through a first pass on your changes to `_dill.py`, and they look very good.  I'm still looking into your use of `sentinel` and `shim` objects, and what would be `dill.nonlocals`.
mmckerns: @tvalentyn: this PR should keep backward compatibility with older pickles.  You may want to try it out and confirm.  @matsjoyce: I'm interested in your thoughts on the `_shims` module.
tvalentyn: Glad to see progress on #300 (should probably also be in the PR title). Thanks, @anivegesana .
@mmckerns these days we prefer to keep a tight bound on dill version in Apache Beam so as to not expect any forwards / backwards compatibility guarantees. 
tvalentyn: cc: @ryanthompson491 FYI.
tvalentyn: cc: @ryanthompson591 FYI.
anivegesana: I am still trying to make `_shims.py` a little bit more readable and understandable. It is confusing because it kind of borders on a separate language on its own that lazily evaluates during unpickling. I am kind of thinking of splitting part of `Shim` off to `GetAttrShim` and making the additions to `_dill` inline instead of at the end. A little more verbose, but hopefully more readable because it provides a better separation of concerns.

Let me know if it made it more confusing instead of helping.
anivegesana: 1. I couldn't think of any other uses of `Shim`, so I looked at `cloudpickle`'s code to see if there is any use in the future. It seems that it could be useful for pickling dictionary views. In their code, they have something that looks like:
```py
@register(type({}.values()))
def save_dict_values(pickler, obj):
    log.info("Dv: <%s>" % (obj,))
    pickler.save_reduce(dict.values, (dict(enumerate(obj)),), obj=obj)
    log.info("# Dv")
```
With `Shim`, it can be written in the following form, which doesn't store the entire dictionary, just the values, and reconstructs a dictionary that looks like it later.
```py
@register(type({}.values()))
def save_dict_values(pickler, obj):
    log.info("Dv: <%s>" % (obj,))
    dict_constructor = _shims.Shim((dicttype, (enumerate(obj),)))
    pickler.save_reduce(dict.values, (dict_constructor),), obj=obj)
    log.info("# Dv")
```

On a sidenote, I can see that they added their own `save_code` that, while it still needs to create a new code object from scratch, doesn't modify the bytecode. We could piggyback off that and remove `_nonlocals.py`. https://github.com/cloudpipe/cloudpickle/blob/5d89947288a18029672596a4d719093cc6d5a412/cloudpickle/cloudpickle.py#L393-L486
2. I think that renaming it to something like `@assign_to(_dill)` seems to make the most sense for this.
3. Based on 1 it may make sense to rename it to `LazyReduce` or something similar. I won't worry about that now, just the doc strings.
mmckerns: 1.  There are, I believe, three or so places that `dill` broke backward compatibility in the past.  Anywhere that's happened, @tvalentyn has pretty much shown up in the comments. It might be worthwhile to check those cases... and PR #406.   I also think a solution like `save_code` from `cloudpickle` is a bit more preferable than what's in `_nonlocals`, because as you say, it doesn't modify the bytecode.
2. Agreed.
3. Agreed. `LazyReduce` might be a better name... and then we wouldn't worry about it being used directly or not as that has a very specific use case.
anivegesana: I am now a little more preferential to `Reduce` because the two lines are supposed to be equivalent:

```py
pickler.save_reduce(func, args)
pickler.save(Reduce(func, args))
```

This actually seems to get us the equivalent of `cloudpickle_fast` for free if we get rid of the hard-coded globals (should be easy, but best to push that off to another release.) 
mmckerns: - Yes, I agree with you `Reduce` is better.
- With regard to `cloudpickle_fast`... that's excellent, and I also agree with you there.  I also agree that it should be a separate issue.  Once this is merged, we should create a separate issue as follow-up, even if it's a placeholder for now.  I'm more than happy to have you continue to contribute. :)
- I had the same thought, `Getattr` is probably better and more consistent.  It could be converted to a function -- I'll have to ponder that a bit more before I know if that's preferable. 
- I like the thinking behind making `Recuce` a tuple... but I'm not sure if or what kind of negative impact that might have.  Often in cases like this, the best thing to do is to try it, and see if you run into any oddness in the creation and "usability" of the object.  An alternate option might be to add a class method that is used to produce a tuple within a call to `__reduce__`.

anivegesana: Alright. Things once again look good for another lookover. I have a couple of things outside the scope of this PR that could probably be useful for this release.

I now have a couple functions from cloudpickle for saving dictionary views that I will open another PR for after this one.

Also, it seems like numpy fixed their pickling by numpy 1.21, so plenty of functions are redundant. I don't know which version each one was implemented, but removing `NumpyUfuncType`, `NumpyDType`, and `NumpyArrayType` from `dump` for numpy 1.21 and later would delegate the pickling to the proper numpy functions. More correct versions can be found for when those three were implemented, but it doesn't seem to be worth the effort currently.
mmckerns: Ok, great. I'll have a look at your changes.  With regard to dictionary views, sure, please do open a new PR. Also, with regard to pickling `numpy` objects, yeah, that probably should be revisited as the logic in `dill` for the objects you mention is quite old and from a time when `numpy` objects didn't always pickle well.  I'd also like to see the close but stalled PR #427 get tied up before a release, but that may not happen (-- care to have a look at it?)

anivegesana: I was looking over the code in ABC, and I believe it could benefit from the same system as we have here. I'll rename _recursive_cells back to _postproc so that it can be used for objects that aren't just cells, such as registered classes for an ABC.
mmckerns: > I was looking over the code in ABC, and I believe it could benefit from the same system as we have here. I'll rename _recursive_cells back to _postproc so that it can be used for objects that aren't just cells, such as registered classes for an ABC.

Awesome.  I thought you might find that issue relevant.  It should be a separate issue aside from the naming of objects and arguments.  I try to minimize any changes to module object names and interfaces, so pondering the names/interfaces a good bit is desirable. 
anivegesana: The current `Reduce`, `Getattr` and `_save_with_postproc` seem to be flexible enough for cPickle and other non-cell manipulations after the creation of any object. I am fine with cementing those names and arguments. Let me know if you have any suggestions.

Thank you for following this PR and wish you a happy new year!
anivegesana: The current `Reduce`, `Getattr` and `_save_with_postproc` seem to be flexible enough for cPickle and other non-cell manipulations after the creation of any object. I am fine with cementing those names and arguments. Let me know if you have any suggestions.

Thank you for following this PR and wish you a happy new year!
mmckerns: @anivegesana: Happy New Year.  Let me know your thoughts on the open comments above.
anivegesana: Fixes #229, fixes #239, fixes #300, fixes #399, fixes #411
anivegesana: There is a slight issue. It seems that the recurse setting does not actually handle recursion. It only finds the minimum set of globals that are needed to run any given function, which is not necessary the same.

```pycon
>>> def f(x):
...   if x<=1:
...     return x
...   else:
...     return f(x-1)+f(x-2)
>>> import dill
>>> dill.settings['recurse']=True
>>> m = dill.dumps(f)
>>> del f
>>> g=dill.loads(m)
>>> g(5)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "<console>", line 5, in f
NameError: name 'f' is not defined
```

My new approach to handle recursion without the 4 almost identical if conditions is the same as cloudpickle's solution (create the function with empty dictionaries and fill them once the function is created), but that solution is incompatible with the current recurse setting. At the same time, recursion cannot be correctly represented without it.

Should this be a new setting?
mmckerns: > Should this be a new setting?
Sure.

Let me know when you want another (final?) review.

mmckerns: FYI: testing with `multiprocess` (master) yields failures in python 3.7 and 3.8:
```
======================================================================
ERROR: test_map_async_callbacks (__main__.WithManagerTestPool)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mmckerns/dev/git/uqfoundation/multiprocess/py3.7/multiprocess/tests/__init__.py", line 2278, in test_map_async_callbacks
    error_callback=call_args.append).wait()
  File "<string>", line 2, in map_async
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess-0.70.13.dev0-py3.7.egg/multiprocess/managers.py", line 818, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess-0.70.13.dev0-py3.7.egg/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess-0.70.13.dev0-py3.7.egg/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill-0.3.5.dev0-py3.7.egg/dill/_dill.py", line 589, in dump
    StockPickler.dump(self, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.7/site-packages/dill-0.3.5.dev0-py3.7.egg/dill/_dill.py", line 1136, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.7/site-packages/dill-0.3.5.dev0-py3.7.egg/dill/_dill.py", line 1321, in save_instancemethod0
    pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.7/site-packages/dill-0.3.5.dev0-py3.7.egg/dill/_dill.py", line 1708, in save_function
    )), obj=obj, postproc_list=postproc_list) #, delayed_objs=(globs)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill-0.3.5.dev0-py3.7.egg/dill/_dill.py", line 513, in _save_with_postproc
    postproc = pickler._postproc.pop(id(obj))
KeyError: 4452114656
```
and python 3.9 and 3.10:
```
======================================================================
ERROR: test_abort (__main__.WithManagerTestBarrier)
Test that an abort will put the barrier in a broken state
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mmckerns/dev/git/uqfoundation/multiprocess/py3.9/multiprocess/tests/__init__.py", line 1856, in test_abort
    self.run_threads(self._test_abort_f,
  File "/Users/mmckerns/dev/git/uqfoundation/multiprocess/py3.9/multiprocess/tests/__init__.py", line 1770, in run_threads
    b = Bunch(self, f, args, self.N-1)
  File "/Users/mmckerns/dev/git/uqfoundation/multiprocess/py3.9/multiprocess/tests/__init__.py", line 1706, in __init__
    p.start()
  File "/Users/mmckerns/lib/python3.9/site-packages/multiprocess-0.70.13.dev0-py3.9.egg/multiprocess/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/mmckerns/lib/python3.9/site-packages/multiprocess-0.70.13.dev0-py3.9.egg/multiprocess/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/mmckerns/lib/python3.9/site-packages/multiprocess-0.70.13.dev0-py3.9.egg/multiprocess/context.py", line 291, in _Popen
    return Popen(process_obj)
  File "/Users/mmckerns/lib/python3.9/site-packages/multiprocess-0.70.13.dev0-py3.9.egg/multiprocess/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
  File "/Users/mmckerns/lib/python3.9/site-packages/multiprocess-0.70.13.dev0-py3.9.egg/multiprocess/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/mmckerns/lib/python3.9/site-packages/multiprocess-0.70.13.dev0-py3.9.egg/multiprocess/popen_forkserver.py", line 47, in _launch
    reduction.dump(process_obj, buf)
  File "/Users/mmckerns/lib/python3.9/site-packages/multiprocess-0.70.13.dev0-py3.9.egg/multiprocess/reduction.py", line 63, in dump
    ForkingPickler(file, protocol, *args, **kwds).dump(obj)
  File "/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py", line 589, in dump
    StockPickler.dump(self, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 487, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py", line 1136, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py", line 1321, in save_instancemethod0
    pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py", line 1136, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py", line 1321, in save_instancemethod0
    pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py", line 1630, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py", line 516, in _save_with_postproc
    pickler.save_reduce(*reduction)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 691, in save_reduce
    save(func)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py", line 1563, in save_module
    pickler.save_global(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 1070, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <module 'dill._dill' from '/Users/mmckerns/lib/python3.9/site-packages/dill-0.3.5.dev0-py3.9.egg/dill/_dill.py'>: it's not found as __main__.dill._dill
```
and also errors with all versions of `pypy`.
anivegesana: I am still looking over a couple of stuff related to `_save_with_preproc` and saving functions.

Using `__setstate__` for functions instead of additional arguments like fdict will be more resilient to recursive structures because they are applied after the object is created. Because of this, we should consider deprecating those additional arguments. The drawback would be that the names of each of these new arguments would be stored in the pickle file as well, which may or may not be worth it in all scenarios.

After some consideration, I noticed was that for functions, this setting is ultimately useless. In the cases that recurse is on, the globals dictionary will be reconstructed for each function. I found that I can optimize a little and then use `__setstate__` or `pickler._batch_setitems` instead of dict.update. Now, turning on this new setting would allow recursion at the expense of just 1-2 PUT instructions for each `__main__` function in most cases, which doesn’t seem like a big overhead. In the case that both settings are off, this is the same as the original functionality. Because the module dicts are pickled by reference, the other two cases can never be enabled (hence the bad code coverage.) Unless there are plans to pickle modules by value (like the experimental feature in cloudpickle) or there is a strong reason to keep it for functions, I will let with ‘recurse’ automatically handle recursion since there is nearly zero overhead.

On the other hand, for classes a new setting to control postprocessing of `__dict__` would solve #381. Since this PR is getting bloated, I’ll leave that for another PR.

Also, thanks for drawing attention the multiprocess library. The current implementation of `_save_with_preproc` doesn’t seem to be thread-safe. I’ll look into condition locks and see if it solves the issue with little overhead.

https://docs.python.org/2.7/library/threading.html#threading.Condition

I’ll let you know when I am done.
anivegesana: Ok. Thank you for your patience.

It looks like the function object in PyPy defines `__setstate__` so the default implementation of the BUILD opcode is overwritten. Unfortunately, it is incompatible with the default, so there is no way to use `__setstate__` for functions in a portable way. Since a recursive `__dict__` attribute on a function is probably rare and it seems to be too difficult to find a solution that works with both CPython and PyPy without resorting to the trick #422 (except with postprocessing steps), I only handled recursive functions themselves (`__globals__` contains the function and `recurse` is on) which is the much more common case and leave the other case for whoever revisits #422.

The issue was not thread safety. Since there is only one stream for each pickler, making the pickler thread safe doesn’t make much sense. The issue was that I forgot a case within the saving of the globals in `save_function`. I had assumed that whenever the globals were specified and recurse was off, it would mean that the globals was a module dictionary, but this wasn’t the case. When this happened, it is possible that a cyclic reference was made in the globals dictionary that I didn’t catch. I needed to choose the object that was first pushed onto the stack as the place where the cycle was broken. This would the earliest place that I could update the globals dictionary with the new globals.

Because of the difficulty of locating this bug, I added warning messages for conditions that dill cannot currently cover or that dill covers with different settings. Let me know if this is unnecessary, and I will remove these warnings. 

I will also run the test cases for the other pathos modules to make sure they all work with the changes to dill that I have made so I can be more confident that there are no more blunders.
mmckerns: Thanks for being vigilant on this PR. I'll check out your new changes.  ...and yeah, I usually test any major changes to `dill` on at least `multiprocess` in my "final" review, just to see if there are any new unexpected issues that the `dill` tests might not catch.
mmckerns: Also, I noticed that a `PicklingWarning` is thrown warning that `The byref setting is on...`, but the invoking code doesn't result in an error, and instead passes the test.  Try the `klepto` tests.
anivegesana: > Also, I noticed that a `PicklingWarning` is thrown warning that `The byref setting is on...`, but the invoking code doesn't result in an error, and instead passes the test. For example, try the `klepto` tests (seen in at least python 3.7). I assumed that the intent of the warning was to be more informative for the error that is expected to follow. However, if the warning is thrown "often" and there's still a pickling "success"... is this a good thing?

The warning was appearing whenever `byref` was used instead of when the object was recursive. It should be fixed now and the message should only show up in cases that cause `UnpicklingError`s when the pickle is loaded (both in current and older versions of dill.)

Patch:
diff --git a/.travis.yml b/.travis.yml
index d1f2183..a07fbed 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,10 +22,10 @@ matrix:
         - python: '3.9'
           env:
 
-        - python: '3.10-dev'
+        - python: '3.10'
           env:
 
-        - python: 'nightly'
+        - python: '3.11-dev'
           env:
 
         - python: 'pypy2.7-7.3.1' # most recent
@@ -43,7 +43,7 @@ matrix:
     allow_failures:
         - python: '2.7'
         - python: 'pypy2.7-7.3.1'
-        - python: 'nightly'
+        - python: '3.11-dev'
     fast_finish: true
 
 cache:
diff --git a/dill/__init__.py b/dill/__init__.py
index fef89ef..690542f 100644
--- a/dill/__init__.py
+++ b/dill/__init__.py
@@ -25,7 +25,8 @@ __license__ = """
 from ._dill import dump, dumps, load, loads, dump_session, load_session, \
     Pickler, Unpickler, register, copy, pickle, pickles, check, \
     HIGHEST_PROTOCOL, DEFAULT_PROTOCOL, PicklingError, UnpicklingError, \
-    HANDLE_FMODE, CONTENTS_FMODE, FILE_FMODE
+    HANDLE_FMODE, CONTENTS_FMODE, FILE_FMODE, PickleError, PickleWarning, \
+    PicklingWarning, UnpicklingWarning
 from . import source, temp, detect
 
 # get global settings
diff --git a/dill/_dill.py b/dill/_dill.py
index 0023958..6e78b83 100644
--- a/dill/_dill.py
+++ b/dill/_dill.py
@@ -18,7 +18,8 @@ Test against CH16+ Std. Lib. ... TBD.
 __all__ = ['dump','dumps','load','loads','dump_session','load_session',
            'Pickler','Unpickler','register','copy','pickle','pickles',
            'check','HIGHEST_PROTOCOL','DEFAULT_PROTOCOL','PicklingError',
-           'UnpicklingError','HANDLE_FMODE','CONTENTS_FMODE','FILE_FMODE']
+           'UnpicklingError','HANDLE_FMODE','CONTENTS_FMODE','FILE_FMODE',
+           'PickleError','PickleWarning','PicklingWarning','UnpicklingWarning']
 
 import logging
 log = logging.getLogger("dill")
@@ -28,8 +29,7 @@ def _trace(boolean):
     if boolean: log.setLevel(logging.INFO)
     else: log.setLevel(logging.WARN)
     return
-
-stack = dict()  # record of 'recursion-sensitive' pickled objects
+import warnings
 
 import os
 import sys
@@ -39,6 +39,7 @@ PY3 = (sys.hexversion >= 0x3000000)
 # OLDER: 3.0 <= x < 3.4 *OR* x < 2.7.10  #NOTE: guessing relevant versions
 OLDER = (PY3 and sys.hexversion < 0x3040000) or (sys.hexversion < 0x2070ab1)
 OLD33 = (sys.hexversion < 0x3030000)
+OLD37 = (sys.hexversion < 0x3070000)
 PY34 = (0x3040000 <= sys.hexversion < 0x3050000)
 if PY3: #XXX: get types from .objtypes ?
     import builtins as __builtin__
@@ -72,7 +73,7 @@ else:
          GeneratorType, DictProxyType, XRangeType, SliceType, TracebackType, \
          NotImplementedType, EllipsisType, FrameType, ModuleType, \
          BufferType, BuiltinMethodType, TypeType
-from pickle import HIGHEST_PROTOCOL, PicklingError, UnpicklingError
+from pickle import HIGHEST_PROTOCOL, PickleError, PicklingError, UnpicklingError
 try:
     from pickle import DEFAULT_PROTOCOL
 except ImportError:
@@ -261,6 +262,34 @@ except NameError:
     except NameError: ExitType = None
     singletontypes = []
 
+import inspect
+
+### Shims for different versions of Python and dill
+class Sentinel(object):
+    """
+    Create a unique sentinel object that is pickled as a constant.
+    """
+    def __init__(self, name, module_name=None):
+        self.name = name
+        if module_name is None:
+            # Use the calling frame's module
+            self.__module__ = inspect.currentframe().f_back.f_globals['__name__']
+        else:
+            self.__module__ = module_name # pragma: no cover
+    def __repr__(self):
+        return self.__module__ + '.' + self.name # pragma: no cover
+    def __copy__(self):
+        return self # pragma: no cover
+    def __deepcopy__(self, memo):
+        return self # pragma: no cover
+    def __reduce__(self):
+        return self.name
+    def __reduce_ex__(self, protocol):
+        return self.name
+
+from . import _shims
+from ._shims import Reduce, Getattr
+
 ### File modes
 #: Pickles the file handle, preserving mode. The position of the unpickled
 #: object is as for a new file handle.
@@ -460,6 +489,14 @@ class MetaCatchingDict(dict):
         else:
             raise KeyError()
 
+class PickleWarning(Warning, PickleError):
+    pass
+
+class PicklingWarning(PickleWarning, PicklingError):
+    pass
+
+class UnpicklingWarning(PickleWarning, UnpicklingError):
+    pass
 
 ### Extend the Picklers
 class Pickler(StockPickler):
@@ -481,9 +518,9 @@ class Pickler(StockPickler):
         self._strictio = False #_strictio
         self._fmode = settings['fmode'] if _fmode is None else _fmode
         self._recurse = settings['recurse'] if _recurse is None else _recurse
+        self._postproc = {}
 
     def dump(self, obj): #NOTE: if settings change, need to update attributes
-        stack.clear()  # clear record of 'recursion-sensitive' pickled objects
         # register if the object is a numpy ufunc
         # thanks to Paul Kienzle for pointing out ufuncs didn't pickle
         if NumpyUfuncType and numpyufunc(obj):
@@ -528,7 +565,6 @@ class Pickler(StockPickler):
             raise PicklingError(msg)
         else:
             StockPickler.dump(self, obj)
-        stack.clear()  # clear record of 'recursion-sensitive' pickled objects
         return
     dump.__doc__ = StockPickler.dump.__doc__
     pass
@@ -680,6 +716,7 @@ def _create_function(fcode, fglobals, fname=None, fdefaults=None,
     # thus we need to make sure that we have __builtins__ as well
     if "__builtins__" not in func.__globals__:
         func.__globals__["__builtins__"] = globals()["__builtins__"]
+    # assert id(fglobals) == id(func.__globals__)
     return func
 
 def _create_code(*args):
@@ -876,12 +913,25 @@ class _attrgetter_helper(object):
             attrs[index] = ".".join([attrs[index], attr])
         return type(self)(attrs, index)
 
+# _CELL_REF and _CELL_EMPTY are used to stay compatible with versions of dill
+# whose _create_cell functions do not have a default value.
+# _CELL_REF can be safely removed entirely (replaced by empty tuples for calls
+# to _create_cell) once breaking changes are allowed.
+_CELL_REF = None
+_CELL_EMPTY = Sentinel('_CELL_EMPTY')
+
 if PY3:
-    def _create_cell(contents):
-        return (lambda y: contents).__closure__[0]
+    def _create_cell(contents=None):
+        if contents is not _CELL_EMPTY:
+            value = contents
+        return (lambda: value).__closure__[0]
+
 else:
-    def _create_cell(contents):
-        return (lambda y: contents).func_closure[0]
+    def _create_cell(contents=None):
+        if contents is not _CELL_EMPTY:
+            value = contents
+        return (lambda: value).func_closure[0]
+
 
 def _create_weakref(obj, *args):
     from weakref import ref
@@ -977,6 +1027,56 @@ def _locate_function(obj, session=False):
     found = _import_module(obj.__module__ + '.' + obj.__name__, safe=True)
     return found is obj
 
+
+def _save_with_postproc(pickler, reduction, is_pickler_dill=None, obj=Getattr.NO_DEFAULT, postproc_list=None):
+    if obj is Getattr.NO_DEFAULT:
+        obj = Reduce(reduction) # pragma: no cover
+
+    if is_pickler_dill is None:
+        is_pickler_dill = is_dill(pickler, child=True)
+    if is_pickler_dill:
+        # assert id(obj) not in pickler._postproc, str(obj) + ' already pushed on stack!'
+        # if not hasattr(pickler, 'x'): pickler.x = 0
+        # print(pickler.x*' ', 'push', obj, id(obj), pickler._recurse)
+        # pickler.x += 1
+        if postproc_list is None:
+            postproc_list = []
+
+        # Recursive object not supported. Default to a global instead.
+        if id(obj) in pickler._postproc:
+            name = '%s.%s ' % (obj.__module__, getattr(obj, '__qualname__', obj.__name__)) if hasattr(obj, '__module__') else ''
+            warnings.warn('Cannot pickle %r: %shas recursive self-references that trigger a RecursionError.' % (obj, name), PicklingWarning)
+            pickler.save_global(obj)
+            return
+        pickler._postproc[id(obj)] = postproc_list
+
+    # TODO: Use state_setter in Python 3.8 to allow for faster cPickle implementations
+    pickler.save_reduce(*reduction, obj=obj)
+
+    if is_pickler_dill:
+        # pickler.x -= 1
+        # print(pickler.x*' ', 'pop', obj, id(obj))
+        postproc = pickler._postproc.pop(id(obj))
+        # assert postproc_list == postproc, 'Stack tampered!'
+        for reduction in reversed(postproc):
+            if reduction[0] is dict.update and type(reduction[1][0]) is dict:
+                # use the internal machinery of pickle.py to speedup when
+                # updating a dictionary in postproc
+                dest, source = reduction[1]
+                if source:
+                    pickler.write(pickler.get(pickler.memo[id(dest)][0]))
+                    pickler._batch_setitems(iter(source.items()))
+                else:
+                    # Updating with an empty dictionary. Same as doing nothing.
+                    continue
+            else:
+                pickler.save_reduce(*reduction)
+            # pop None created by calling preprocessing step off stack
+            if PY3:
+                pickler.write(bytes('0', 'UTF-8'))
+            else:
+                pickler.write('0')
+
 #@register(CodeType)
 #def save_code(pickler, obj):
 #    log.info("Co: %s" % obj)
@@ -1067,7 +1167,6 @@ def save_module_dict(pickler, obj):
 
 @register(ClassType)
 def save_classobj(pickler, obj): #FIXME: enable pickler._byref
-   #stack[id(obj)] = len(stack), obj
     if obj.__module__ == '__main__': #XXX: use _main_module.__name__ everywhere?
         log.info("C1: %s" % obj)
         pickler.save_reduce(ClassType, (obj.__name__, obj.__bases__,
@@ -1280,6 +1379,10 @@ if sys.hexversion >= 0x20500f0:
     @register(MethodWrapperType)
     def save_instancemethod(pickler, obj):
         log.info("Mw: %s" % obj)
+        if IS_PYPY2 and obj.__self__ is None and obj.im_class:
+            # Can be a class method in PYPY2 if __self__ is none
+            pickler.save_reduce(getattr, (obj.im_class, obj.__name__), obj=obj)
+            return
         pickler.save_reduce(getattr, (obj.__self__, obj.__name__), obj=obj)
         log.info("# Mw")
         return
@@ -1296,10 +1399,44 @@ elif not IS_PYPY:
 
 @register(CellType)
 def save_cell(pickler, obj):
-    log.info("Ce: %s" % obj)
-    f = obj.cell_contents
+    try:
+        f = obj.cell_contents
+    except:
+        log.info("Ce3: %s" % obj)
+        # _shims._CELL_EMPTY is defined in _shims.py to support PyPy 2.7.
+        # It unpickles to a sentinel object _dill._CELL_EMPTY, also created in
+        # _shims.py. This object is not present in Python 3 because the cell's
+        # contents can be deleted in newer versions of Python. The reduce object
+        # will instead unpickle to None if unpickled in Python 3.
+
+        # When breaking changes are made to dill, (_shims._CELL_EMPTY,) can
+        # be replaced by () OR the delattr function can be removed repending on
+        # whichever is more convienient.
+        pickler.save_reduce(_create_cell, (_shims._CELL_EMPTY,), obj=obj)
+        # Call the function _delattr on the cell's cell_contents attribute
+        # The result of this function call will be None
+        pickler.save_reduce(_shims._delattr, (obj, 'cell_contents'))
+        # pop None created by calling _delattr off stack
+        if PY3:
+            pickler.write(bytes('0', 'UTF-8'))
+        else:
+            pickler.write('0')
+        log.info("# Ce3")
+        return
+    if is_dill(pickler, child=True):
+        postproc = pickler._postproc.get(id(f))
+        if postproc is not None:
+            log.info("Ce2: %s" % obj)
+            # _CELL_REF is defined in _shims.py to support older versions of
+            # dill. When breaking changes are made to dill, (_CELL_REF,) can
+            # be replaced by ()
+            postproc.append((_shims._setattr, (obj, 'cell_contents', f)))
+            pickler.save_reduce(_create_cell, (_CELL_REF,), obj=obj)
+            log.info("# Ce2")
+            return
+    log.info("Ce1: %s" % obj)
     pickler.save_reduce(_create_cell, (f,), obj=obj)
-    log.info("# Ce")
+    log.info("# Ce1")
     return
 
 if not IS_PYPY:
@@ -1412,22 +1549,22 @@ def save_weakproxy(pickler, obj):
 @register(ModuleType)
 def save_module(pickler, obj):
     if False: #_use_diff:
-        if obj.__name__ != "dill":
+        if obj.__name__.split('.', 1)[0] != "dill":
             try:
                 changed = diff.whats_changed(obj, seen=pickler._diff_cache)[0]
             except RuntimeError:  # not memorised module, probably part of dill
                 pass
             else:
-                log.info("M1: %s with diff" % obj)
+                log.info("M2: %s with diff" % obj)
                 log.info("Diff: %s", changed.keys())
                 pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
                                     state=changed)
-                log.info("# M1")
+                log.info("# M2")
                 return
 
-        log.info("M2: %s" % obj)
+        log.info("M1: %s" % obj)
         pickler.save_reduce(_import_module, (obj.__name__,), obj=obj)
-        log.info("# M2")
+        log.info("# M1")
     else:
         # if a module file name starts with prefix, it should be a builtin
         # module, so should be pickled as a reference
@@ -1440,7 +1577,7 @@ def save_module(pickler, obj):
                            'site-packages' in obj.__file__)
         else:
             builtin_mod = True
-        if obj.__name__ not in ("builtins", "dill") \
+        if obj.__name__ not in ("builtins", "dill", "dill._dill") \
            and not builtin_mod or is_dill(pickler, child=True) and obj is pickler._main:
             log.info("M1: %s" % obj)
             _main_dict = obj.__dict__.copy() #XXX: better no copy? option to copy?
@@ -1449,6 +1586,10 @@ def save_module(pickler, obj):
             pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
                                 state=_main_dict)
             log.info("# M1")
+        elif PY3 and obj.__name__ == "dill._dill":
+            log.info("M2: %s" % obj)
+            pickler.save_global(obj, name="_dill")
+            log.info("# M2")
         else:
             log.info("M2: %s" % obj)
             pickler.save_reduce(_import_module, (obj.__name__,), obj=obj)
@@ -1457,8 +1598,7 @@ def save_module(pickler, obj):
     return
 
 @register(TypeType)
-def save_type(pickler, obj):
-   #stack[id(obj)] = len(stack), obj #XXX: probably don't obj in all cases below
+def save_type(pickler, obj, postproc_list=None):
     if obj in _typemap:
         log.info("T1: %s" % obj)
         pickler.save_reduce(_load_type, (_typemap[obj],), obj=obj)
@@ -1469,33 +1609,7 @@ def save_type(pickler, obj):
         pickler.save_reduce(_create_namedtuple, (getattr(obj, "__qualname__", obj.__name__), obj._fields, obj.__module__), obj=obj)
         log.info("# T6")
         return
-    elif obj.__module__ == '__main__':
-        if issubclass(type(obj), type):
-        #   try: # used when pickling the class as code (or the interpreter)
-            if is_dill(pickler, child=True) and not pickler._byref:
-                # thanks to Tom Stepleton pointing out pickler._session unneeded
-                _t = 'T2'
-                log.info("%s: %s" % (_t, obj))
-                _dict = _dict_from_dictproxy(obj.__dict__)
-        #   except: # punt to StockPickler (pickle by class reference)
-            else:
-                log.info("T5: %s" % obj)
-                name = getattr(obj, '__qualname__', getattr(obj, '__name__', None))
-                StockPickler.save_global(pickler, obj, name=name)
-                log.info("# T5")
-                return
-        else:
-            _t = 'T3'
-            log.info("%s: %s" % (_t, obj))
-            _dict = obj.__dict__
-       #print (_dict)
-       #print ("%s\n%s" % (type(obj), obj.__name__))
-       #print ("%s\n%s" % (obj.__bases__, obj.__dict__))
-        for name in _dict.get("__slots__", []):
-            del _dict[name]
-        pickler.save_reduce(_create_type, (type(obj), obj.__name__,
-                                           obj.__bases__, _dict), obj=obj)
-        log.info("# %s" % _t)
+
     # special cases: NoneType, NotImplementedType, EllipsisType
     elif obj is type(None):
         log.info("T7: %s" % obj)
@@ -1513,16 +1627,50 @@ def save_type(pickler, obj):
         log.info("T7: %s" % obj)
         pickler.save_reduce(type, (Ellipsis,), obj=obj)
         log.info("# T7")
+
     else:
-        log.info("T4: %s" % obj)
-       #print (obj.__dict__)
-       #print ("%s\n%s" % (type(obj), obj.__name__))
-       #print ("%s\n%s" % (obj.__bases__, obj.__dict__))
-        name = getattr(obj, '__qualname__', getattr(obj, '__name__', None))
-        StockPickler.save_global(pickler, obj, name=name)
-        log.info("# T4")
+        obj_name = getattr(obj, '__qualname__', getattr(obj, '__name__', None))
+        _byref = getattr(pickler, '_byref', None)
+        obj_recursive = id(obj) in getattr(pickler, '_postproc', ())
+        incorrectly_named = not _locate_function(obj)
+        if not _byref and not obj_recursive and incorrectly_named: # not a function, but the name was held over
+            if issubclass(type(obj), type):
+                # thanks to Tom Stepleton pointing out pickler._session unneeded
+                _t = 'T2'
+                log.info("%s: %s" % (_t, obj))
+                _dict = _dict_from_dictproxy(obj.__dict__)
+            else:
+                _t = 'T3'
+                log.info("%s: %s" % (_t, obj))
+                _dict = obj.__dict__
+           #print (_dict)
+           #print ("%s\n%s" % (type(obj), obj.__name__))
+           #print ("%s\n%s" % (obj.__bases__, obj.__dict__))
+            for name in _dict.get("__slots__", []):
+                del _dict[name]
+            _save_with_postproc(pickler, (_create_type, (
+                type(obj), obj_name, obj.__bases__, _dict
+            )), obj=obj, postproc_list=postproc_list)
+            log.info("# %s" % _t)
+        else:
+            log.info("T4: %s" % obj)
+            if incorrectly_named:
+                warnings.warn('Cannot locate reference to %r.' % (obj,), PicklingWarning)
+            if obj_recursive:
+                warnings.warn('Cannot pickle %r: %s.%s has recursive self-references that trigger a RecursionError.' % (obj, obj.__module__, obj_name), PicklingWarning)
+           #print (obj.__dict__)
+           #print ("%s\n%s" % (type(obj), obj.__name__))
+           #print ("%s\n%s" % (obj.__bases__, obj.__dict__))
+            StockPickler.save_global(pickler, obj, name=obj_name)
+            log.info("# T4")
     return
 
+# Error in PyPy 2.7 when adding ABC support
+if IS_PYPY2:
+    @register(FrameType)
+    def save_frame(pickler, obj):
+        raise PicklingError('Cannot pickle a Python stack frame')
+
 @register(property)
 def save_property(pickler, obj):
     log.info("Pr: %s" % obj)
@@ -1541,6 +1689,18 @@ def save_classmethod(pickler, obj):
         orig_func = obj.__get__(None, object)
         if isinstance(obj, classmethod):
             orig_func = getattr(orig_func, im_func) # Unbind
+
+    # if PY3:
+    #     if type(obj.__dict__) is dict:
+    #         if obj.__dict__:
+    #             state = obj.__dict__
+    #         else:
+    #             state = None
+    #     else:
+    #         state = (None, {'__dict__', obj.__dict__})
+    # else:
+    #     state = None
+
     pickler.save_reduce(type(obj), (orig_func,), obj=obj)
     log.info("# Cm")
 
@@ -1548,53 +1708,56 @@ def save_classmethod(pickler, obj):
 def save_function(pickler, obj):
     if not _locate_function(obj): #, pickler._session):
         log.info("F1: %s" % obj)
-        if getattr(pickler, '_recurse', False):
+        _recurse = getattr(pickler, '_recurse', None)
+        _byref = getattr(pickler, '_byref', None)
+        _postproc = getattr(pickler, '_postproc', None)
+        postproc_list = []
+        if _recurse:
             # recurse to get all globals referred to by obj
             from .detect import globalvars
-            globs = globalvars(obj, recurse=True, builtin=True)
-            # remove objects that have already been serialized
-           #stacktypes = (ClassType, TypeType, FunctionType)
-           #for key,value in list(globs.items()):
-           #    if isinstance(value, stacktypes) and id(value) in stack:
-           #        del globs[key]
-            # ABORT: if self-references, use _recurse=False
-            if id(obj) in stack: # or obj in globs.values():
-                globs = obj.__globals__ if PY3 else obj.func_globals
+            globs_copy = globalvars(obj, recurse=True, builtin=True)
+
+            # Add the name of the module to the globs dictionary to prevent
+            # the duplication of the dictionary. Pickle the unpopulated
+            # globals dictionary and set the remaining items after the function
+            # is created to correctly handle recursion.
+            globs = {'__name__': obj.__module__}
         else:
-            globs = obj.__globals__ if PY3 else obj.func_globals
-        _byref = getattr(pickler, '_byref', None)
-        _recurse = getattr(pickler, '_recurse', None)
-        _memo = (id(obj) in stack) and (_recurse is not None)
-       #print("stack: %s + '%s'" % (set(hex(i) for i in stack),hex(id(obj))))
-        stack[id(obj)] = len(stack), obj
+            globs_copy = obj.__globals__ if PY3 else obj.func_globals
+
+            # If the globals is a module __dict__, do not save it in the pickle.
+            if globs_copy is not None and obj.__module__ is not None and \
+                    getattr(_import_module(obj.__module__, True), '__dict__', None) is globs_copy:
+                globs = globs_copy
+            else:
+                globs = {'__name__': obj.__module__}
+
+        if globs_copy is not None and globs is not globs_copy:
+            # In the case that the globals are copied, we need to ensure that
+            # the globals dictionary is updated when all objects in the
+            # dictionary are already created.
+            if PY3:
+                glob_ids = {id(g) for g in globs_copy.values()}
+            else:
+                glob_ids = {id(g) for g in globs_copy.itervalues()}
+            for stack_element in _postproc:
+                if stack_element in glob_ids:
+                    _postproc[stack_element].append((dict.update, (globs, globs_copy)))
+                    break
+            else:
+                postproc_list.append((dict.update, (globs, globs_copy)))
+
         if PY3:
-            #NOTE: workaround for 'super' (see issue #75)
-            _super = ('super' in getattr(obj.__code__,'co_names',())) and (_byref is not None)
-            if _super: pickler._byref = True
-            if _memo: pickler._recurse = False
             fkwdefaults = getattr(obj, '__kwdefaults__', None)
-            pickler.save_reduce(_create_function, (obj.__code__,
-                                globs, obj.__name__,
-                                obj.__defaults__, obj.__closure__,
-                                obj.__dict__, fkwdefaults), obj=obj)
+            _save_with_postproc(pickler, (_create_function, (
+                  obj.__code__, globs, obj.__name__, obj.__defaults__,
+                  obj.__closure__, obj.__dict__, fkwdefaults
+            )), obj=obj, postproc_list=postproc_list)
         else:
-            _super = ('super' in getattr(obj.func_code,'co_names',())) and (_byref is not None) and getattr(pickler, '_recurse', False)
-            if _super: pickler._byref = True
-            if _memo: pickler._recurse = False
-            pickler.save_reduce(_create_function, (obj.func_code,
-                                globs, obj.func_name,
-                                obj.func_defaults, obj.func_closure,
-                                obj.__dict__), obj=obj)
-        if _super: pickler._byref = _byref
-        if _memo: pickler._recurse = _recurse
-       #clear = (_byref, _super, _recurse, _memo)
-       #print(clear + (OLDER,))
-        #NOTE: workaround for #234; "partial" still is problematic for recurse
-        if OLDER and not _byref and (_super or (not _super and _memo) or (not _super and not _memo and _recurse)): pickler.clear_memo()
-       #if _memo:
-       #    stack.remove(id(obj))
-       #   #pickler.clear_memo()
-       #   #StockPickler.clear_memo(pickler)
+            _save_with_postproc(pickler, (_create_function, (
+                obj.func_code, globs, obj.func_name, obj.func_defaults,
+                obj.func_closure, obj.__dict__
+            )), obj=obj, postproc_list=postproc_list)
         log.info("# F1")
     else:
         log.info("F2: %s" % obj)
@@ -1626,7 +1789,6 @@ def pickles(obj,exact=False,safe=False,**kwds):
             #FIXME: should be "(pik == obj).all()" for numpy comparison, though that'll fail if shapes differ
             result = bool(pik.all() == obj.all())
         except AttributeError:
-            import warnings
             warnings.filterwarnings('ignore')
             result = pik == obj
             warnings.resetwarnings()
diff --git a/dill/_shims.py b/dill/_shims.py
new file mode 100644
index 0000000..ac20eca
--- /dev/null
+++ b/dill/_shims.py
@@ -0,0 +1,266 @@
+#!/usr/bin/env python
+#
+# Author: Mike McKerns (mmckerns @caltech and @uqfoundation)
+# Author: Anirudh Vegesana (avegesan@stanford.edu)
+# Copyright (c) 2021 The Uncertainty Quantification Foundation.
+# License: 3-clause BSD.  The full license text is available at:
+#  - https://github.com/uqfoundation/dill/blob/master/LICENSE
+"""
+Provides shims for compatibility between versions of dill and Python.
+
+Compatibility shims should be provided in this file. Here are two simple example
+use cases.
+
+Deprecation of constructor function:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Assume that we were transitioning _import_module in _dill.py to
+the builtin function importlib.import_module when present.
+
+@move_to(_dill)
+def _import_module(import_name):
+    ... # code already in _dill.py
+
+_import_module = Getattr(importlib, 'import_module', Getattr(_dill, '_import_module', None))
+
+The code will attempt to find import_module in the importlib module. If not
+present, it will use the _import_module function in _dill.
+
+Emulate new Python behavior in older Python versions:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+CellType.cell_contents behaves differently in Python 3.6 and 3.7. It is
+read-only in Python 3.6 and writable and deletable in 3.7.
+
+if _dill.OLD37 and _dill.HAS_CTYPES and ...:
+    @move_to(_dill)
+    def _setattr(object, name, value):
+        if type(object) is _dill.CellType and name == 'cell_contents':
+            _PyCell_Set.argtypes = (ctypes.py_object, ctypes.py_object)
+            _PyCell_Set(object, value)
+        else:
+            setattr(object, name, value)
+... # more cases below
+
+_setattr = Getattr(_dill, '_setattr', setattr)
+
+_dill._setattr will be used when present to emulate Python 3.7 functionality in
+older versions of Python while defaulting to the standard setattr in 3.7+.
+
+See this PR for the discussion that lead to this system:
+https://github.com/uqfoundation/dill/pull/443
+"""
+
+import inspect, sys
+
+_dill = sys.modules['dill._dill']
+
+
+class Reduce(object):
+    """
+    Reduce objects are wrappers used for compatibility enforcement during
+    unpickle-time. They should only be used in calls to pickler.save and
+    other Reduce objects. They are only evaluated within unpickler.load.
+
+    Pickling a Reduce object makes the two implementations equivalent:
+
+    pickler.save(Reduce(*reduction))
+
+    pickler.save_reduce(*reduction, obj=reduction)
+    """
+    __slots__ = ['reduction']
+    def __new__(cls, *reduction, **kwargs):
+        """
+        Args:
+            *reduction: a tuple that matches the format given here:
+              https://docs.python.org/3/library/pickle.html#object.__reduce__
+            is_callable: a bool to indicate that the object created by
+              unpickling `reduction` is callable. If true, the current Reduce
+              is allowed to be used as the function in further save_reduce calls
+              or Reduce objects.
+        """
+        is_callable = kwargs.get('is_callable', False) # Pleases Py2. Can be removed later
+        if is_callable:
+            self = object.__new__(_CallableReduce)
+        else:
+            self = object.__new__(Reduce)
+        self.reduction = reduction
+        return self
+    def __repr__(self):
+        return 'Reduce%s' % (self.reduction,)
+    def __copy__(self):
+        return self # pragma: no cover
+    def __deepcopy__(self, memo):
+        return self # pragma: no cover
+    def __reduce__(self):
+        return self.reduction
+    def __reduce_ex__(self, protocol):
+        return self.__reduce__()
+
+class _CallableReduce(Reduce):
+    # A version of Reduce for functions. Used to trick pickler.save_reduce into
+    # thinking that Reduce objects of functions are themselves meaningful functions.
+    def __call__(self, *args, **kwargs):
+        reduction = self.__reduce__()
+        func = reduction[0]
+        f_args = reduction[1]
+        obj = func(*f_args)
+        return obj(*args, **kwargs)
+
+__NO_DEFAULT = _dill.Sentinel('Getattr.NO_DEFAULT')
+
+def Getattr(object, name, default=__NO_DEFAULT):
+    """
+    A Reduce object that represents the getattr operation. When unpickled, the
+    Getattr will access an attribute 'name' of 'object' and return the value
+    stored there. If the attribute doesn't exist, the default value will be
+    returned if present.
+
+    The following statements are equivalent:
+
+    Getattr(collections, 'OrderedDict')
+    Getattr(collections, 'spam', None)
+    Getattr(*args)
+
+    Reduce(getattr, (collections, 'OrderedDict'))
+    Reduce(getattr, (collections, 'spam', None))
+    Reduce(getattr, args)
+
+    During unpickling, the first two will result in collections.OrderedDict and
+    None respectively because the first attribute exists and the second one does
+    not, forcing it to use the default value given in the third argument.
+    """
+
+    if default is Getattr.NO_DEFAULT:
+        reduction = (getattr, (object, name))
+    else:
+        reduction = (getattr, (object, name, default))
+
+    return Reduce(*reduction, is_callable=callable(default))
+
+Getattr.NO_DEFAULT = __NO_DEFAULT
+del __NO_DEFAULT
+
+def move_to(module, name=None):
+    def decorator(func):
+        if name is None:
+            fname = func.__name__
+        else:
+            fname = name
+        module.__dict__[fname] = func
+        func.__module__ = module.__name__
+        return func
+    return decorator
+
+######################
+## Compatibility Shims are defined below
+######################
+
+_CELL_EMPTY = Getattr(_dill, '_CELL_EMPTY', None)
+
+if _dill.OLD37:
+    if _dill.HAS_CTYPES and hasattr(_dill.ctypes, 'pythonapi') and hasattr(_dill.ctypes.pythonapi, 'PyCell_Set'):
+        # CPython
+        ctypes = _dill.ctypes
+
+        _PyCell_Set = ctypes.pythonapi.PyCell_Set
+
+        @move_to(_dill)
+        def _setattr(object, name, value):
+            if type(object) is _dill.CellType and name == 'cell_contents':
+                _PyCell_Set.argtypes = (ctypes.py_object, ctypes.py_object)
+                _PyCell_Set(object, value)
+            else:
+                setattr(object, name, value)
+
+        @move_to(_dill)
+        def _delattr(object, name):
+            if type(object) is _dill.CellType and name == 'cell_contents':
+                _PyCell_Set.argtypes = (ctypes.py_object, ctypes.c_void_p)
+                _PyCell_Set(object, None)
+            else:
+                delattr(object, name)
+
+    # General Python (not CPython) up to 3.6 is in a weird case, where it is
+    # possible to pickle recursive cells, but we can't assign directly to the
+    # cell.
+    elif _dill.PY3:
+        # Use nonlocal variables to reassign the cell value.
+        # https://stackoverflow.com/a/59276835
+        __nonlocal = ('nonlocal cell',)
+        exec('''def _setattr(cell, name, value):
+            if type(cell) is _dill.CellType and name == 'cell_contents':
+                def cell_setter(value):
+                    %s
+                    cell = value # pylint: disable=unused-variable
+                func = _dill.FunctionType(cell_setter.__code__, globals(), "", None, (cell,)) # same as cell_setter, but with cell being the cell's contents
+                func(value)
+            else:
+                setattr(cell, name, value)''' % __nonlocal)
+        move_to(_dill)(_setattr)
+
+        exec('''def _delattr(cell, name):
+            if type(cell) is _dill.CellType and name == 'cell_contents':
+                try:
+                    cell.cell_contents
+                except:
+                    return
+                def cell_deleter():
+                    %s
+                    del cell # pylint: disable=unused-variable
+                func = _dill.FunctionType(cell_deleter.__code__, globals(), "", None, (cell,)) # same as cell_deleter, but with cell being the cell's contents
+                func()
+            else:
+                delattr(cell, name)''' % __nonlocal)
+        move_to(_dill)(_delattr)
+
+    else:
+        # Likely PyPy 2.7. Simulate the nonlocal keyword with bytecode
+        # manipulation.
+
+        # The following function is based on 'cell_set' from 'cloudpickle'
+        # https://github.com/cloudpipe/cloudpickle/blob/5d89947288a18029672596a4d719093cc6d5a412/cloudpickle/cloudpickle.py#L393-L482
+        # Copyright (c) 2012, Regents of the University of California.
+        # Copyright (c) 2009 `PiCloud, Inc. <http://www.picloud.com>`_.
+        # License: https://github.com/cloudpipe/cloudpickle/blob/master/LICENSE
+        @move_to(_dill)
+        def _setattr(cell, name, value):
+            if type(cell) is _dill.CellType and name == 'cell_contents':
+                _cell_set = _dill.FunctionType(
+                      _cell_set_template_code, {}, '_cell_set', (), (cell,),)
+                _cell_set(value)
+            else:
+                setattr(cell, name, value)
+
+        def _cell_set_factory(value):
+            lambda: cell
+            cell = value
+
+        co = _cell_set_factory.__code__
+
+        _cell_set_template_code = _dill.CodeType(
+            co.co_argcount,
+            co.co_nlocals,
+            co.co_stacksize,
+            co.co_flags,
+            co.co_code,
+            co.co_consts,
+            co.co_names,
+            co.co_varnames,
+            co.co_filename,
+            co.co_name,
+            co.co_firstlineno,
+            co.co_lnotab,
+            co.co_cellvars,  # co_freevars is initialized with co_cellvars
+            (),  # co_cellvars is made empty
+        )
+
+        del co
+
+        @move_to(_dill)
+        def _delattr(cell, name):
+            if type(cell) is _dill.CellType and name == 'cell_contents':
+                pass
+            else:
+                delattr(cell, name)
+
+_setattr = Getattr(_dill, '_setattr', setattr)
+_delattr = Getattr(_dill, '_delattr', delattr)
diff --git a/dill/detect.py b/dill/detect.py
index a4c23f6..2ceccbe 100644
--- a/dill/detect.py
+++ b/dill/detect.py
@@ -157,7 +157,16 @@ def freevars(func):
         func = getattr(func, func_code).co_freevars # get freevars
     else:
         return {}
-    return dict((name,c.cell_contents) for (name,c) in zip(func,closures))
+
+    def get_cell_contents():
+        for (name,c) in zip(func,closures):
+            try:
+                cell_contents = c.cell_contents
+            except:
+                continue
+            yield (name,c.cell_contents)
+
+    return dict(get_cell_contents())
 
 # thanks to Davies Liu for recursion of globals
 def nestedglobals(func, recurse=True):
@@ -201,9 +210,14 @@ def globalvars(func, recurse=True, builtin=False):
         # get references from within closure
         orig_func, func = func, set()
         for obj in getattr(orig_func, func_closure) or {}:
-            _vars = globalvars(obj.cell_contents, recurse, builtin) or {}
-            func.update(_vars) #XXX: (above) be wary of infinte recursion?
-            globs.update(_vars)
+            try:
+                cell_contents = obj.cell_contents
+            except:
+                pass
+            else:
+                _vars = globalvars(cell_contents, recurse, builtin) or {}
+                func.update(_vars) #XXX: (above) be wary of infinte recursion?
+                globs.update(_vars)
         # get globals
         globs.update(getattr(orig_func, func_globals) or {})
         # get names of references
diff --git a/tox.ini b/tox.ini
index 102690d..8c7f65f 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,9 +12,11 @@ envlist =
     py38
     py39
     py310
+    py311
 #   pypy27
     pypy36
     pypy37
+    pypy38
 
 [testenv]
 deps =


Test patch:
diff --git a/tests/test_classdef.py b/tests/test_classdef.py
index 56ad0ce..d59f6f6 100644
--- a/tests/test_classdef.py
+++ b/tests/test_classdef.py
@@ -85,8 +85,10 @@ def test_class_objects():
             assert type(_cls).__name__ == "_meta"
 
 # test NoneType
-def test_none():
+def test_specialtypes():
     assert dill.pickles(type(None))
+    assert dill.pickles(type(NotImplemented))
+    assert dill.pickles(type(Ellipsis))
 
 if hex(sys.hexversion) >= '0x20600f0':
     from collections import namedtuple
@@ -204,7 +206,7 @@ def test_slots():
 if __name__ == '__main__':
     test_class_instances()
     test_class_objects()
-    test_none()
+    test_specialtypes()
     test_namedtuple()
     test_dtype()
     test_array_nested()
diff --git a/tests/test_functions.py b/tests/test_functions.py
index 40f4c99..696801c 100644
--- a/tests/test_functions.py
+++ b/tests/test_functions.py
@@ -36,6 +36,12 @@ def function_e(e, *e1, e2=1, e3=2):
     return e + sum(e1) + e2 + e3''')
 
 
+def function_with_unassigned_variable():
+    if False:
+        value = None
+    return (lambda: value)
+
+
 def test_functions():
     dumped_func_a = dill.dumps(function_a)
     assert dill.loads(dumped_func_a)(0) == 0
@@ -52,6 +58,17 @@ def test_functions():
     assert dill.loads(dumped_func_d)(1, 2, 3) == 6
     assert dill.loads(dumped_func_d)(1, 2, d2=3) == 6
 
+    empty_cell = function_with_unassigned_variable()
+    cell_copy = dill.loads(dill.dumps(empty_cell))
+    assert 'empty' in str(cell_copy.__closure__[0])
+    try:
+        cell_copy()
+    except:
+        # this is good
+        pass
+    else:
+        raise AssertionError('cell_copy() did not read an empty cell')
+
     if is_py3():
         exec('''
 dumped_func_e = dill.dumps(function_e)
@@ -62,6 +79,5 @@ assert dill.loads(dumped_func_e)(1, 2, e2=3, e3=4) == 10
 assert dill.loads(dumped_func_e)(1, 2, 3, e2=4) == 12
 assert dill.loads(dumped_func_e)(1, 2, 3, e2=4, e3=5) == 15''')
 
-
 if __name__ == '__main__':
     test_functions()
diff --git a/tests/test_recursive.py b/tests/test_recursive.py
index af8ce53..ef779fe 100644
--- a/tests/test_recursive.py
+++ b/tests/test_recursive.py
@@ -6,9 +6,27 @@
 #  - https://github.com/uqfoundation/dill/blob/master/LICENSE
 
 import dill
+from dill._dill import PY3
 from functools import partial
-from dill._dill import PY3, OLDER
-_super = super
+import warnings
+
+
+def copy(obj, byref=False, recurse=False):
+    if byref:
+        try:
+            return dill.copy(obj, byref=byref, recurse=recurse)
+        except:
+            pass
+        else:
+            raise AssertionError('Copy of %s with byref=True should have given a warning!' % (obj,))
+
+        warnings.simplefilter('ignore')
+        val = dill.copy(obj, byref=byref, recurse=recurse)
+        warnings.simplefilter('error')
+        return val
+    else:
+        return dill.copy(obj, byref=byref, recurse=recurse)
+
 
 class obj1(object):
     def __init__(self):
@@ -16,7 +34,7 @@ class obj1(object):
 
 class obj2(object):
     def __init__(self):
-        _super(obj2, self).__init__()
+        super(obj2, self).__init__()
 
 class obj3(object):
     super_ = super
@@ -25,20 +43,20 @@ class obj3(object):
 
 
 def test_super():
-    assert dill.copy(obj1(), byref=True)
-    assert dill.copy(obj1(), byref=True, recurse=True)
-   #assert dill.copy(obj1(), recurse=True) #FIXME: fails __main__.py
-    assert dill.copy(obj1())
+    assert copy(obj1(), byref=True)
+    assert copy(obj1(), byref=True, recurse=True)
+    assert copy(obj1(), recurse=True)
+    assert copy(obj1())
 
-    assert dill.copy(obj2(), byref=True)
-    assert dill.copy(obj2(), byref=True, recurse=True)
-   #assert dill.copy(obj2(), recurse=True) #FIXME: fails __main__.py
-    assert dill.copy(obj2())
+    assert copy(obj2(), byref=True)
+    assert copy(obj2(), byref=True, recurse=True)
+    assert copy(obj2(), recurse=True)
+    assert copy(obj2())
 
-    assert dill.copy(obj3(), byref=True)
-    assert dill.copy(obj3(), byref=True, recurse=True)
-   #assert dill.copy(obj3(), recurse=True) #FIXME: fails __main__.py
-    assert dill.copy(obj3())
+    assert copy(obj3(), byref=True)
+    assert copy(obj3(), byref=True, recurse=True)
+    assert copy(obj3(), recurse=True)
+    assert copy(obj3())
 
 
 def get_trigger(model):
@@ -56,11 +74,10 @@ class Model(object):
 
 
 def test_partial():
-    assert dill.copy(Machine(), byref=True)
-    assert dill.copy(Machine(), byref=True, recurse=True)
-    if not OLDER:
-        assert dill.copy(Machine(), recurse=True)
-    assert dill.copy(Machine())
+    assert copy(Machine(), byref=True)
+    assert copy(Machine(), byref=True, recurse=True)
+    assert copy(Machine(), recurse=True)
+    assert copy(Machine())
 
 
 class Machine2(object):
@@ -72,21 +89,77 @@ class Machine2(object):
 
 class SubMachine(Machine2):
     def __init__(self):
-        _super(SubMachine, self).__init__()
-        #super(SubMachine, self).__init__() #XXX: works, except for 3.1-3.3
+        super(SubMachine, self).__init__()
 
 
 def test_partials():
-    assert dill.copy(SubMachine(), byref=True)
-    assert dill.copy(SubMachine(), byref=True, recurse=True)
-   #if not OLDER: #FIXME: fails __main__.py
-   #    assert dill.copy(SubMachine(), recurse=True)
-    assert dill.copy(SubMachine())
+    assert copy(SubMachine(), byref=True)
+    assert copy(SubMachine(), byref=True, recurse=True)
+    assert copy(SubMachine(), recurse=True)
+    assert copy(SubMachine())
+
 
+class obj4(object):
+    def __init__(self):
+        super(obj4, self).__init__()
+        a = self
+        class obj5(object):
+            def __init__(self):
+                super(obj5, self).__init__()
+                self.a = a
+        self.b = obj5()
+
+
+def test_circular_reference():
+    assert copy(obj4())
+    obj4_copy = dill.loads(dill.dumps(obj4()))
+    if PY3:
+        assert type(obj4_copy) is type(obj4_copy).__init__.__closure__[0].cell_contents
+        assert type(obj4_copy.b) is type(obj4_copy.b).__init__.__closure__[0].cell_contents
+
+
+def f():
+    def g():
+       return g
+    return g
+
+
+def test_function_cells():
+    assert copy(f())
+
+
+def fib(n):
+    assert n >= 0
+    if n <= 1:
+        return n
+    else:
+        return fib(n-1) + fib(n-2)
+
+
+def test_recursive_function():
+    global fib
+    fib2 = copy(fib, recurse=True)
+    fib3 = copy(fib)
+    fib4 = fib
+    del fib
+    assert fib2(5) == 5
+    for _fib in (fib3, fib4):
+        try:
+            _fib(5)
+        except:
+            # This is expected to fail because fib no longer exists
+            pass
+        else:
+            raise AssertionError("Function fib shouldn't have been found")
+    fib = fib4
 
 
 if __name__ == '__main__':
-    #print(('byref','_super','_recurse','_memo','_stop','OLDER'))
-    test_super()
-    test_partial()
-    test_partials()
+    with warnings.catch_warnings():
+        warnings.simplefilter('error')
+        test_super()
+        test_partial()
+        test_partials()
+        test_circular_reference()
+        test_function_cells()
+        test_recursive_function()
diff --git a/tests/test_selected.py b/tests/test_selected.py
index de3877a..bd79083 100644
--- a/tests/test_selected.py
+++ b/tests/test_selected.py
@@ -79,7 +79,7 @@ def test_frame_related():
   _is = lambda ok: not ok if dill._dill.IS_PYPY2 else ok
   ok = dill.pickles(f)
   if verbose: print ("%s: %s, %s" % (ok, type(f), f))
-  assert _is(not ok) #XXX: dill fails
+  assert not ok
   ok = dill.pickles(g)
   if verbose: print ("%s: %s, %s" % (ok, type(g), g))
   assert _is(not ok) #XXX: dill fails