Repository: ipython/ipython

Issue:
reduce client caching of results
The current parallel code doesn't _need_ to store much in memory.  But it does, particularly in the Client.  We cache results for convenience in the Client's `results` and `metadata` dicts.  This data is readily reproducible from the the Hub, so discarding it beyond a certain point shouldn't be an issue.

To easily run your client out of memory, simply echo a large array a number of times:

``` python
A = np.zeros(1024*1024)
for i in range(100):
    rc[0].apply_sync(lambda x:x, A)
```

And bam, you have 100 copies of A in memory.

We should figure out a sensible scheme for purging old results, so the client doesn't grow too much during data-intensive tasks.

One thing we could do is have AsyncResult objects _remove_ results from the Client's result dict when they are retrieved.  It's true that it's possible to have multiple AsyncResults pointing to the same message, but it's not likely, and shouldn't come up in normal use.



Discussion:
Another, more expensive idea would be to hash incoming results, and deduplicate the contents of the cache.  While this would improve performance of my test cases dramatically, it probably wouldn't have much effect on real-world code.

> One thing we could do is have AsyncResult objects remove results from 
> the Client's result dict when they are retrieved.

That would be nice, especially as Client.results.clear() cannot safely 
be used while iterating over an AsyncResult.

```
from IPython.parallel import Client
c = Client()
lv = c.load_balanced_view()

@lv.parallel(ordered=False)
def g(i, s):
    import time
    import os
    time.sleep(s)
    return i, os.getpid()

a = g.map(range(10), [0.1] * 10)
for i in a:
        print i
        c.results.clear()

(0, 7520)
(6, 7016)
None
None
None
None
None
None
(8, 7520)
(9, 4820)
```

It seems that this is not only a caching problem:

```
task = []
res = []
for a in batches: # 20 batches
        for x in vals: # starts 800 new tasks
                tasks.append(lv.apply(...))                
        for task in tasks:
            res.extend(task.get()) # Each task returns a list of dicts: 50 dicts a 18 elements         
        import asizeof  # from http://code.activestate.com/recipes/546530/
        print ("in lv= " +str(asizeof.asizeof(lv))) # prints size of lv in bytes
        print ("in c= " +str(asizeof.asizeof(c)))
        lv.purge_results("all") # does nothing for the size of lv?
        print ("in lv= " +str(asizeof.asizeof(lv)))
        print ("in c= " +str(asizeof.asizeof(c)))
        lv.results.clear() # again nothing
        print ("in lv= " +str(asizeof.asizeof(lv)))
        print ("in c= " +str(asizeof.asizeof(c)))
        lv.client.results.clear() # frees a lot
        print ("in lv= " +str(asizeof.asizeof(lv)))
        print ("in c= " +str(asizeof.asizeof(c)))
        lv.client.metadata.clear() # frees some more, but still more as in the last round
        tasks = []
        print ("out lv= " +str(asizeof.asizeof(lv)))
        print ("out c= " +str(asizeof.asizeof(c)))
        print("res= " + str(asizeof.asizeof(res)))
        print("batch done"
```

Results in the following output:

```
Saving: 1/20 batches
in lv= 54067416
in c= 54024640
in lv= 54067488
in c= 54024712
in lv= 54067488
in c= 54024712
in lv= 2612456
in c= 2569680
out lv= 470208
out c= 427352
res= 51285320
batch done
in lv= 54310224
in c= 54261496
in lv= 54310296
in c= 54261568
in lv= 54310296
in c= 54261568
in lv= 2855264
in c= 2806536
out lv= 712584
out c= 663776
res= 102580072
batch done
[... skipping 17 batches...]
in lv= 60679952
in c= 60503648
in lv= 60680024
in c= 60503720
in lv= 60680024
in c= 60503720
in lv= 9224992
in c= 9048688
out lv= 7082312
out c= 6905928
res= 1026169976
batch done
```

So with 16.000 tasks the client grows from 430kb to 6MB.

A few points:
- purge_results is a purely remote call, there is no reason for it to affect local caches
- view.results should _be_ client.results, but there is a bug in traitlets preventing this for views created before the first result is retrieved (see #2311)

There are two other objects that grow as a function of the number of messages, but only matter if you have quite a large number of messages (they are not a function of the size of messages, only the number):
- `Client.history` is a list of msg_ids that the client has submitted (each View also has a history)
- `Client.session.digest_history` is a `set` of digests used for authentication (prevents snoops from sending duplicate messages)

I think these are the only remaining stateful structures, so a _complete_ reset of a Client should be:

``` python
def clear_client(rc):
    assert not rc.outstanding, "Can't clear a client with outstanding tasks!"
    rc.results.clear()
    rc.metadata.clear()
    rc.history = []
    rc.session.digest_history.clear()
```

and `view.history = []` for the View's history as well.

With these calls, you should be able to get permanent zero-growth (which I have confirmed with an adaptation of your  script).

I wouldn't be _too_ concerned about the history/digests, because it really does take a large amount of tasks for them to grow.  A quick asizeof check suggests that you should get about 3000 tasks per MB, which means you need to submit a quarter million before hitting 100 MB.  If you are submitting that many, then I suppose you do need to stay on top of these.

Thanks a lot for all the explanations!

Would you pull such an addition to `Client`? 

If I understand the code right, a "non-caching" `Client` would also be possible if the `AsyncResult` in [line 154](https://github.com/ipython/ipython/blob/master/IPython/parallel/client/asyncresult.py#L154) instead of only getting the results would also remove the `msg_ids` from both `self._client.results` and `self._client.metadata`?

So something along the lines of 

```
results = map(self._client.results.get, self.msg_ids)
if self._client._remove_from_cache:
    for id in self.msg_ids:
        del self._client.results[id]
        del self._client.metadata[id]
```

yes, pull requests are certainly welcome.  The model I have in mind is a boolean 'pop' property of AsyncResults that determines whether they would use pop or get from the Client's structures.  It might need some careful thought to get right in all cases.

I think it's appropriate for the history/digest_history to be clearable via a public method, but they should _not_ be cleared under normal circumstances.  Possibly there should be a 'local' flag to client.purge_results.

Huih, the Code in `Client` and `AsyncResult` is a lot harder to understand than I thought... I also found a few minor bugs -> see a first [commit](https://github.com/JanSchulz/ipython/commits/client-clear-cache), which _only_ adresses the "purge all cached values" but not the "pop instead of get" case. 

(Is it better to already send a pull request for such "unready" code?)

I also found some discrepancies between the docstring of `client.get_result()` and `client.wait()` and the actual code:

``` python
from IPython.parallel import Client 
c = Client(profile="laptop-only")  
lv = c.load_balanced_view()  

def do_work(input=None):
    return input
tasks = []

for i in range(10):
    tasks.append(lv.apply(do_work, input=i))  

for task in tasks:
    print(str(task.get()))

lv.client.results.clear()
lv.client.metadata.clear()
lv.client.history = []
lv.client.session.digest_history.clear()
lv.results.clear()

# Now "redownload all the results from the hub
tasks_renew = []

for task in tasks:
    tasks_renew.append(c.get_result(task.msg_ids))

c.wait()
# Actually I would expect that all the results are now available   
# But from the code the tasks in tasks_renew are all AsyncHubResults and 
# therefore only resubmitted on task.get().
# So this will print both zero:
print(len(c.results))
print(len(c.outstanding))
```

Wouldn't it make more sense to just let `client.get_results()` just resubmit all not locally available tasks (not in `client.outstanding` or `client.result`) and then return a AsyncResult directly? The code from `status_result()`could be refactored into a `_reget_from_hub(msg_ids)`.

On Sat, Aug 25, 2012 at 2:53 PM, JanSchulz notifications@github.com wrote:

> Huih, the Code in Client and AsyncResult is a lot harder to understand
> than I thought... I also found a few minor bugs -> see a first commithttps://github.com/JanSchulz/ipython/commits/client-clear-cache,
> which _only_ adresses the "purge all cached values" but not the "pop
> instead of get" case.
> 
> I suspect the 'pop instead of get' bit will be far from trivial (this is
> why I haven't done it already!).
> 
> (Is it better to already send a pull request for such "unready" code?)

A PR is totally appropriate as soon as you have code worth talking about.
 It doesn't need to be merge-ready.

> I also found some discrepancies between the docstring of
> client.get_result() and client.wait() and the actual code:
> 
> from IPython.parallel import Client c = Client(profile="laptop-only")  lv = c.load_balanced_view()
> def do_work(input=None):
>     return inputtasks = []
> for i in range(10):
>     tasks.append(lv.apply(do_work, input=i))
> for task in tasks:
>     print(str(task.get()))
> lv.client.results.clear()lv.client.metadata.clear()lv.client.history = []lv.client.session.digest_history.clear()lv.results.clear()
> 
> # Now "redownload all the results from the hubtasks_renew = []
> 
> for task in tasks:
>     tasks_renew.append(c.get_result(task.msg_ids))
> c.wait()# Actually I would expect that all the results are now available   # But from the code the tasks in tasks_renew are all AsyncHubResults and # therefore only resubmitted on task.get().
> 
> Nothing is resubmitted - AsyncHubResults only poll for _results_.

There are two fixes for the discrepancy in client.wait() from what you are
expecting:
1. update the docstring, so it is clear that client.wait() only checks for
   outstanding tasks _that it submitted_.  This is the current behavior.
2. when creating an AsyncHubResult, add to a parallel
   `outstanding_from_hub` set, so that client.wait() polls for these as well
   as regular outstanding tasks.

Waiting on AsyncHubResults is quite expensive, so I might lean towards 1 -
that way HubResults are only awaited _explicitly_.  But maybe it makes
sense, to avoid confusion like you faced.  AsyncHubResults are sufficiently
uncommon to use, that it's probably okay.

> # So this will print both zero:print(len(c.results))print(len(c.outstanding))
> 
> Wouldn't it make more sense to just let client.get_results() just
> resubmit all not locally available tasks (not in client.outstanding or
> client.result) and then return a AsyncResult directly? The code from
> status_result()could be refactored into a _reget_from_hub(msg_ids).
> 
> Absolutely not.  Asking to recompute something that may have taken hours or
> days when the result is sitting in the Hub's database would be silly, and
> extraordinarily wasteful.  It also requires the rarely satisfied assumption
> that all tasks are atomic, and not dependent on the state of the engines.
> 
>  —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/1131#issuecomment-8027854.

> There are two fixes for the discrepancy in client.wait() from what you are
> expecting:
> [...]

Actually I would expect that they are submitted to the hub on `get_result()` and just become available on `wait()`, just like normal tasks. If I would use `time.sleep(...)` instead of `client.wait()` normal tasks would be ready (assuming more wait time than execution time) and iterating over them would be instant but with AsyncHubResult they just get submitted while iterating which in worst case (hundreds of AsyncHubResults with one msg_id each, no results cached on the hub) would mean they are serialized.

> Absolutely not.  Asking to recompute something that may have taken hours or
> days when the result is sitting in the Hub's database would be silly, and
> extraordinarily wasteful.  It also requires the rarely satisfied assumption
> that all tasks are atomic, and not dependent on the state of the engines.

Actually I don't want `_reget_from_hub()` to recompute it. From my understanding of the whole `AsyncHubResult` mechanism and the `client.purge_results()` method, it just gets the result from the hub cache (I haven't looked into the Controller: what is cached there)? 

If so, I don't get why waiting on `AsnycHubResult` is more expensive than waiting on `AsyncResult`: best case the result is in the hub cache; worst case it's the same as if resubmitting a new task and waiting for that.

Or did I miss something?

On Sat, Aug 25, 2012 at 4:00 PM, JanSchulz notifications@github.com wrote:

>  There are two fixes for the discrepancy in client.wait() from what you
> are
> expecting:
> [...]
> 
>  Actually I would expect that they are submitted to the hub on
> get_result() and just become available on wait(), just like normal tasks.
> If I would use time.sleep(...) instead of client.wait() normal tasks
> would be ready (assuming more wait time than execution time) and iterating
> over them would be instant but with AsyncHubResult they just get submitted
> while iterating which in worst case (hundreds of AsyncHubResults with one
> msg_id each, no results cached on the hub) would mean they are serialized.
> 
> Ah, sorry, I was confused - you keep saying 'resubmit', which has a very
> specific meaning wrt IPython Client/Controller, and is apparently not what
> you mean by it.  See below for what 'resubmit' means in this context.
> 
>  Absolutely not. Asking to recompute something that may have taken hours
> or
> days when the result is sitting in the Hub's database would be silly, and
> extraordinarily wasteful. It also requires the rarely satisfied assumption
> that all tasks are atomic, and not dependent on the state of the engines.
> 
> Actually I don't want _reget_from_hub() to recompute it. From my
> understanding of the whole AsyncHubResult mechanism and the
> client.purge_results() method, it just gets the result from the hub cache
> (I haven't looked into the Controller: what is cached there)?
> 
> The hub caches all requests and results of tasks (assuming the Task DB is
> enabled), so they can be resubmitted or retrieved at a later point.  The
> AsyncHubResult is the API for the retrieval.  client.resubmit() does the
> resubmission.
> 
>  If so, I don't get why waiting on AsnycHubResult is more expensive than
> waiting on AsyncResult: best case the result: it's in the hub cache;
> worst case it's the same as if resubmitting a new task and waiting for that.
> 
> Waiting for AsyncHubResults that are not yet done is _totally different_
> from waiting for a regular AsyncResult from an Engine (this is the whole
> reason they are different classes).  This is what is so much more expensive.
> 
> Or did I miss something?
> 
> There are two ways to get results:
1. you submitted a request for execution (AsyncResult).  In this case,
   there will be a reply coming in on the same socket that requested the
   computation.  The only work you do for a 'wait' is simply poll on that zmq
   socket.  It is very cheap, with a cost in the nanoseconds.
2. You did not request that result (or you discarded it), in which case you
   request it from the Hub (AsyncHubResult).  Requesting a result from the hub
   involves making 1-many 'result_status' requests of the Hub.  If you request
   the result and it is not ready, you have to keep submitting
   `result_requests` until it is.  It's the actual waiting that is more
   expensive, because unlike simply polling on a zmq socket, you are polling
   the Hub with a complete message request/reply. This is at least a few
   orders of magnitude more expensive than simply polling on a socket.

The critical difference: Client interactions with Engines are async.
 Client interactions with the Hub are _not_ async.

>  —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/1131#issuecomment-8028312.

Hm, this is rather unfortunate: isn't there a way to make the "re-receive content" request just tell the hub to resend the results for `msg_ids` again over the normal channels and on the client side handle it as in  `send_apply_request`? This would make this whole thing _much_ easier: Then there is no difference to the normal `apply_request`: the wait would simple wait until it is available in `results` (Meaning AsyncHubResult can go away) and `status_result()` could even loose the `status_only` parameter and the corresponding code for the `status_only==False` case.

The Client-Hub protocoll would then be:

send_apply_request -> tell the hub to compute the result for (msg_ids, functions) and send the result back to me (async)
re-send_result_request -> tell the hub to resend the result for msg_ids (or an Exception) (async)
resubmit_request -> tell the hub to recompute a rsult for msg_ids and send it back to me (async)
status -> ask the hub for a status in certain msg_ids (not ansync)

The AsyncResult could then simple wait until all result for it's msg_ids are available in `client.results` and then get (or pop) them to the `AsyncResult._result`

No, the Hub is not involved in apply requests at all. all those go directly to engines via the Schedulers.  The Hub does not have access to the channels used for execution.

@JanSchulz I think your experience would be improved in most cases if `ar.wait(0)` were added to [get_result](https://github.com/ipython/ipython/blob/master/IPython/parallel/client/client.py#L1396) when it's a HubResult.

@minrk: Yep, that would be the easiest way. 

Ok, I just had a closer look how "telling the hub" works and I think I actually meant "the part on the controller responsible for receiving messages from the client". As far as I understand the code on the client side, everything goes to the same socket: `self._query_socket` is used for `purge_results`, `reslt_status` and `resubmit`. So the hub/scheduler/hub should have access to everything the client sent? 

Anyway: the hub has access to everything which is sent back to the client (engine results are cached on the hub): so why not implement the hub side so that "results" are always sent via the same stream/socket instead of once via the query one and once via the one used to pass on the engine results? 

As far as I understand the code it would mean:
- let "status only" be handled by a new message "status_request" (instead of a part of "result_request") and "status_reply".
- splitting the `hub.get_results()` function in two (`get_status()` and `get_results()`) and using two different sockets/streams to send the "result" and the "status". 
- On the client side it would mean to split `client.result_status()` also in two (status-only part and `client.reget_from_hub()`), and change the code in `get_result()` to add the "regetted" msg_ids to `client._outstanding` and then use the new `client.reget_from_hub()` when results are not locally cached and return a normal `AsyncResult`.

This would make lots of code on the client side much easier: `AsyncHubResult` could simple be deleted and `c.result_status()` would lose the code duplication with `_handle_apply_reply()`/`_handle_apply_reply()` and would have a cleaner interface ("Status only" instead of both status and results).

> As far as I understand the code on the client side, everything goes to the same socket: `self._query_socket` is used for `purge_results`, `result_status` and `resubmit`.

With the critical exception, of course, that messages associated with execution (`apply`, `execute`, `abort`) are sent via the Schedulers, and are not seen by the Hub until they have passed through the Scheduler.  The query socket is used for making queries of the cluster state, which resides on the Hub.  This is the sole function of the Hub - to monitor the state of the cluster (perhaps StateMonitor would have been a better name :), and handle queries thereof.

> Anyway:why not implement the hub side so that "results" are always sent via the same stream/socket instead of once via the query one and once via the one used to pass on the engine results?

Because the Hub simply does not have access to the Scheduler sockets on which the original results arrived (those are in different processes, and potentially on different physical machines).

I think your proposal for splitting result_status and get_results still makes good sense - the logic for having both implementations interwoven is gross, and I should have split them as soon as I saw that, rather than slowly adding to it.  But you cannot get rid of AsyncHubResult as you have proposed without the following addition to Hub.get_result:
- when get_result is called and a result is not available, remember the client_id and msg_id in a table.
- Every time results arrive, check the missed result request table, and if it's been requested, send it along.

#2340 gives better manual hooks for this, but doesn't get the automatic cache reduction part necessary to close this issue.

Hi,
I am having trouble with the memory build up, both in the client and on the engines. I wonder whether you could suggest a set of commands that will sort this out. I have tried with *.purge_results('all') and *results.clear() with little success.
thanks in advance for your help,
Into

How many tasks and how big? It can be tricky to make sure that you actually do clear everything on the Client side.

How are you running tasks on the Engine side? Is it exclusively through apply/etc, or do you do a lot of execute/%px?  apply should be fairly stateless.

In general, it helps to summarize your tasks: how many, how frequently, how much data do you move around, what APIs you use the most, etc.

Hi, thanks for the reply. Sorry for not providing enough info, I am quite new to python and more so to iPython parallel.
- How many tasks and how big?

I am using ipython parallel as part of a [MCMC](http://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo) sampler, each iteration has in the order of 10s to 100s tasks and I use in the order of 1000s of iterations. 
- How are you running tasks on the Engine side?

In general I send to the engines an object which encapsulates my data and additional information necessary to perform the jobs. 
- Is it exclusively through apply/etc, or do you do a lot of execute/%px? 

I am using map with block=True on a load-balanced view. I only use execute to load modules on the engines.

The data itself are numpy arrays with integers of variable length, from 100s to 10000s elements each. I normally have about 10s to 100 of this arrays. The data is encapsulated on a object that has the data itself plus some other variables related to the MCMC. it is useful to pass the whole object because it is self contained in terms of the actual computations.

After each iteration, i.e., after running each batch of the tasks I do c.purge_results('all'), where c is the client. The memory usage goes up quite quickly into 10-15 GB ...

To do honest, I do not even know how to diagnose what is using so much memory. I do know that if I close the ipython shell a good bunch of the memory (~1/3) is free and then the rest when I close the ipython cluster.

Thanks a lot in advance for your help,
Inti

Hi, I am not sure If I provided enough info on the previous meesage. Please let me know if there is something I am missing and that I can say to help in getting some help. BW, Inti

Sorry, this one dropped off my radar.  Can you provide an actual code sample to reproduce the issue?  I have found `purge_results('all')` ( or `purge_everything()`, in the case of _very_ large numbers of tasks) to be effective at keeping the cache down, as long as I am not also storing the results elsewhere.

Hi,
I'll try to provide some code. the parallele tasks are part of a larger 
piece of code which may not be easy to reproduce at the moment outside 
my computer. Give me a couple of days to put together a reproduce 
example. I'll try the |purge_everything() call and see how that goes.
thanks !
Inti
|
On 30/05/13 00:06, Min RK wrote:

> Sorry, this one dropped off my radar. Can you provide an actual code 
> sample to reproduce the issue? I have found |purge_results('all')| ( 
> or |purge_everything()|, in the case of /very/ large numbers of tasks) 
> to be effective at keeping the cache down, as long as I am not also 
> storing the results elsewhere.
> 
> —
> Reply to this email directly or view it on GitHub 
> https://github.com/ipython/ipython/issues/1131#issuecomment-18652044.

Has there been any progress on this.  I'm using IPython 2.0 and am trying use map_async on a load balanced view.  However, my parameter study is somewhat large - roughly 25,000 and the memory used by the clients steadily increases and crashes my maching after processing 1,000 of the 25,000 calls needed.  I was able to rewrite my functions using the ProcessPoolExecuter and the memory for each process stays pegged ad 0.8% of my system memory.  I would prefer to use the IPython.parallel infrastructure.

The function that I'm mapping takes 10-15 arguments but produces no output (i.e. all output is dumped to a file)

@pbrady large requests shouldn't be cached in the client, so only results should make the Client memory really grow. What version of pyzmq are you using?

looks like version 14.1.0

$ conda list

# packages in environment at /home/ptb/miniconda3:

#
backports.ssl-match-hostname 3.4.0.2                   <pip>
conda                     3.4.2                    py34_0
conda-build               1.3.2                    py34_0
cython                    0.20.1                   py34_0
dateutil                  2.1                      py34_2
freetype                  2.4.10                        0
ipython                   2.0.0                    py34_0
ipython-notebook          2.0.0                    py34_0
jinja2                    2.7.2                    py34_0
libpng                    1.5.13                        1
libsodium                 0.4.5                         0
llvm                      3.3                           0
llvmpy                    0.12.4                   py34_0
markupsafe                0.18                     py34_0
matplotlib                1.3.1                np18py34_0
mpld3                     0.2                       <pip>
numba                     0.13.1               np18py34_0
numexpr                   2.3.1                np18py34_0
numpy                     1.8.1                    py34_0
openssl                   1.0.1g                        0
pandas                    0.13.1               np18py34_0
patchelf                  0.6                           0
patsy                     0.2.1                np18py34_0
pip                       1.5.4                    py34_0
pycosat                   0.6.0                    py34_0
pygments                  1.6                      py34_0
pyparsing                 2.0.1                    py34_0
python                    3.4.0                         0
python-dateutil           2.1                       <pip>
pytz                      2014.2                   py34_0
pyyaml                    3.11                     py34_0
pyzmq                     14.1.0                   py34_0
readline                  6.2                           2
scipy                     0.13.3               np18py34_0
setuptools                3.4.4                    py34_0
six                       1.6.1                    py34_0
sqlite                    3.8.4.1                       0
ssl_match_hostname        3.4.0.2                  py34_0
statsmodels               0.5.0                np18py34_0
sympy                     0.7.5                    py34_0
system                    5.8                           1
tk                        8.5.13                        0
tornado                   3.2.0                    py34_0
util-linux                2.21                          0
yaml                      0.1.4                         0
zeromq                    4.0.4                         0
zlib                      1.2.7                         0

On Fri, May 2, 2014 at 2:26 PM, Min RK notifications@github.com wrote:

> @pbrady https://github.com/pbrady large requests shouldn't be cached in
> the client, so only results should make the Client memory really grow. What
> version of pyzmq are you using?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/1131#issuecomment-42075392
> .

Try `conda update pyzmq`. There is a memory leak in pyzmq 14.0-14.1, fixed in 14.2 (current on conda).

I update pyzmq but still see the same memory growth.  Here's the relevant
portion of the code, AFAIK

``` python
from IPython.parallel import Client
rc = Client()
dv = rc[:]

h = symbols('h')
ident = 'NonCons'
file_set = simulation.file_set(ident)
opt_dict =
{'interior':cons_dict['int_eq'],'lboundary':cons_dict['lboundary'],

'rboundary':cons_dict['rboundary'],'material_props':{'conv':0.9,'diff':0.01},
            'file_ident':ident,'numeric':True,'file_set':file_set}
# get function ready for async parallel execution
eig = lambda args :
 simulation.eig_analysis(npoints=args[1],ib_frac=args[0],
                                             sympy_subs=args[2],**opt_dict)
dv.execute('import simulation')
dv['opt_dict'] = opt_dict
ib_frac = [1.0/2**i for i in range(20)]
#nx = [20*2**i for i in range(4)]
nx = [40]
e0_subs = list((-1e3/h)/2**i for i in range(18))
e0_subs.extend(list((1e3/h)/2**i for i in range(18)))
d1_subs = list((-1e3/h)/2**i for i in range(18))
d1_subs.extend(list((1e3/h)/2**i for i in range(18)))

subs = ( ((e0,e),(d1,d)) for e in e0_subs for d in d1_subs)
lview = rc.load_balanced_view()
ar = lview.map_async(eig,(p for p in
product(zip(repeat(nx[-1]),ib_frac),nx,subs)) )
```

Am I calling map_async in an incorrect/inefficient way?

Thanks for the quick replies

On Fri, May 2, 2014 at 2:29 PM, Min RK notifications@github.com wrote:

> Try conda update pyzmq. There is a memory leak in pyzmq 14.0-14.1, fixed
> in 14.2 (current on conda).
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/1131#issuecomment-42075631
> .

I'll dig into it.​

So I'm not sure what I changed but now I see the same memory growth using
the ProcessPoolExecuter and even running in serial so I don't think this is
an IPython issue.  Any ideas on how to debug memory leaks in python?

On Fri, May 2, 2014 at 3:28 PM, Min RK notifications@github.com wrote:

> I'll dig into it.​
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/1131#issuecomment-42080958
> .


Resolving patch:
diff --git a/IPython/parallel/client/asyncresult.py b/IPython/parallel/client/asyncresult.py
--- a/IPython/parallel/client/asyncresult.py
+++ b/IPython/parallel/client/asyncresult.py
@@ -16,16 +16,10 @@
 from IPython.parallel import error
 from IPython.utils.py3compat import string_types
 
-#-----------------------------------------------------------------------------
-# Functions
-#-----------------------------------------------------------------------------
 
 def _raw_text(s):
     display_pretty(s, raw=True)
 
-#-----------------------------------------------------------------------------
-# Classes
-#-----------------------------------------------------------------------------
 
 # global empty tracker that's always done:
 finished_tracker = MessageTracker()
@@ -48,8 +42,11 @@ class AsyncResult(object):
     _targets = None
     _tracker = None
     _single_result = False
+    owner = False,
 
-    def __init__(self, client, msg_ids, fname='unknown', targets=None, tracker=None):
+    def __init__(self, client, msg_ids, fname='unknown', targets=None, tracker=None,
+        owner=False,
+    ):
         if isinstance(msg_ids, string_types):
             # always a list
             msg_ids = [msg_ids]
@@ -64,6 +61,7 @@ def __init__(self, client, msg_ids, fname='unknown', targets=None, tracker=None)
         self._fname=fname
         self._targets = targets
         self._tracker = tracker
+        self.owner = owner
         
         self._ready = False
         self._outputs_ready = False
@@ -150,6 +148,12 @@ def wait(self, timeout=-1):
                     # cutoff infinite wait at 10s
                     timeout = 10
                 self._wait_for_outputs(timeout)
+                
+                if self.owner:
+                    
+                    self._metadata = [self._client.metadata.pop(mid) for mid in self.msg_ids]
+                    [self._client.results.pop(mid) for mid in self.msg_ids]
+                
 
 
     def successful(self):
@@ -395,7 +399,7 @@ def _republish_displaypub(self, content, eid):
             return
         md = content['metadata'] or {}
         md['engine'] = eid
-        ip.display_pub.publish(content['source'], content['data'], md)
+        ip.display_pub.publish(data=content['data'], metadata=md)
     
     def _display_stream(self, text, prefix='', file=None):
         if not text:
@@ -691,5 +695,9 @@ def wait(self, timeout=-1):
                 self._success = True
             finally:
                 self._metadata = [self._client.metadata[mid] for mid in self.msg_ids]
+                if self.owner:
+                    [self._client.metadata.pop(mid) for mid in self.msg_ids]
+                    [self._client.results.pop(mid) for mid in self.msg_ids]
+            
 
 __all__ = ['AsyncResult', 'AsyncMapResult', 'AsyncHubResult']
diff --git a/IPython/parallel/client/client.py b/IPython/parallel/client/client.py
--- a/IPython/parallel/client/client.py
+++ b/IPython/parallel/client/client.py
@@ -1358,7 +1358,7 @@ def direct_view(self, targets='all'):
     #--------------------------------------------------------------------------
 
     @spin_first
-    def get_result(self, indices_or_msg_ids=None, block=None):
+    def get_result(self, indices_or_msg_ids=None, block=None, owner=True):
         """Retrieve a result by msg_id or history index, wrapped in an AsyncResult object.
 
         If the client already has the results, no request to the Hub will be made.
@@ -1384,6 +1384,11 @@ def get_result(self, indices_or_msg_ids=None, block=None):
 
         block : bool
             Whether to wait for the result to be done
+        owner : bool [default: True]
+            Whether this AsyncResult should own the result.
+            If so, calling `ar.get()` will remove data from the
+            client's result and metadata cache.
+            There should only be one owner of any given msg_id.
 
         Returns
         -------
@@ -1421,9 +1426,9 @@ def get_result(self, indices_or_msg_ids=None, block=None):
             theids = theids[0]
 
         if remote_ids:
-            ar = AsyncHubResult(self, msg_ids=theids)
+            ar = AsyncHubResult(self, msg_ids=theids, owner=owner)
         else:
-            ar = AsyncResult(self, msg_ids=theids)
+            ar = AsyncResult(self, msg_ids=theids, owner=owner)
 
         if block:
             ar.wait()
@@ -1703,8 +1708,8 @@ def purge_local_results(self, jobs=[], targets=[]):
             if still_outstanding:
                 raise RuntimeError("Can't purge outstanding tasks: %s" % still_outstanding)
             for mid in msg_ids:
-                self.results.pop(mid)
-                self.metadata.pop(mid)
+                self.results.pop(mid, None)
+                self.metadata.pop(mid, None)
 
 
     @spin_first
diff --git a/IPython/parallel/client/view.py b/IPython/parallel/client/view.py
--- a/IPython/parallel/client/view.py
+++ b/IPython/parallel/client/view.py
@@ -1,20 +1,9 @@
-"""Views of remote engines.
+"""Views of remote engines."""
 
-Authors:
+# Copyright (c) IPython Development Team.
+# Distributed under the terms of the Modified BSD License.
 
-* Min RK
-"""
 from __future__ import print_function
-#-----------------------------------------------------------------------------
-#  Copyright (C) 2010-2011  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-----------------------------------------------------------------------------
-
-#-----------------------------------------------------------------------------
-# Imports
-#-----------------------------------------------------------------------------
 
 import imp
 import sys
@@ -315,7 +304,7 @@ def shutdown(self, targets=None, restart=False, hub=False, block=None):
         return self.client.shutdown(targets=targets, restart=restart, hub=hub, block=block)
 
     @spin_after
-    def get_result(self, indices_or_msg_ids=None):
+    def get_result(self, indices_or_msg_ids=None, block=None, owner=True):
         """return one or more results, specified by history index or msg_id.
 
         See :meth:`IPython.parallel.client.client.Client.get_result` for details.
@@ -330,7 +319,7 @@ def get_result(self, indices_or_msg_ids=None):
             for i,index in enumerate(indices_or_msg_ids):
                 if isinstance(index, int):
                     indices_or_msg_ids[i] = self.history[index]
-        return self.client.get_result(indices_or_msg_ids)
+        return self.client.get_result(indices_or_msg_ids, block=block, owner=owner)
 
     #-------------------------------------------------------------------
     # Map
@@ -577,7 +566,9 @@ def _really_apply(self, f, args=None, kwargs=None, targets=None, block=None, tra
         if isinstance(targets, int):
             msg_ids = msg_ids[0]
         tracker = None if track is False else zmq.MessageTracker(*trackers)
-        ar = AsyncResult(self.client, msg_ids, fname=getname(f), targets=_targets, tracker=tracker)
+        ar = AsyncResult(self.client, msg_ids, fname=getname(f), targets=_targets,
+            tracker=tracker, owner=True,
+        )
         if block:
             try:
                 return ar.get()
@@ -656,7 +647,7 @@ def execute(self, code, silent=True, targets=None, block=None):
             msg_ids.append(msg['header']['msg_id'])
         if isinstance(targets, int):
             msg_ids = msg_ids[0]
-        ar = AsyncResult(self.client, msg_ids, fname='execute', targets=_targets)
+        ar = AsyncResult(self.client, msg_ids, fname='execute', targets=_targets, owner=True)
         if block:
             try:
                 ar.get()
@@ -774,7 +765,9 @@ def scatter(self, key, seq, dist='b', flatten=False, targets=None, block=None, t
         else:
             tracker = None
 
-        r = AsyncResult(self.client, msg_ids, fname='scatter', targets=targets, tracker=tracker)
+        r = AsyncResult(self.client, msg_ids, fname='scatter', targets=targets,
+            tracker=tracker, owner=True,
+        )
         if block:
             r.wait()
         else:
@@ -1057,8 +1050,9 @@ def _really_apply(self, f, args=None, kwargs=None, block=None, track=None,
                                 metadata=metadata)
         tracker = None if track is False else msg['tracker']
 
-        ar = AsyncResult(self.client, msg['header']['msg_id'], fname=getname(f), targets=None, tracker=tracker)
-
+        ar = AsyncResult(self.client, msg['header']['msg_id'], fname=getname(f),
+            targets=None, tracker=tracker, owner=True,
+        )
         if block:
             try:
                 return ar.get()


Test patch:
diff --git a/IPython/parallel/tests/test_asyncresult.py b/IPython/parallel/tests/test_asyncresult.py
--- a/IPython/parallel/tests/test_asyncresult.py
+++ b/IPython/parallel/tests/test_asyncresult.py
@@ -1,20 +1,7 @@
-"""Tests for asyncresult.py
+"""Tests for asyncresult.py"""
 
-Authors:
-
-* Min RK
-"""
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2011  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
-
-#-------------------------------------------------------------------------------
-# Imports
-#-------------------------------------------------------------------------------
+# Copyright (c) IPython Development Team.
+# Distributed under the terms of the Modified BSD License.
 
 import time
 
@@ -323,5 +310,33 @@ def single_engine(*a, **kw):
             ar = dv.apply_async(lambda : 5)
             self.assertEqual(ar.get(10), [5])
         self.client._build_targets = save_build
+    
+    def test_owner_pop(self):
+        self.minimum_engines(1)
+        
+        view = self.client[-1]
+        ar = view.apply_async(lambda : 1)
+        ar.get()
+        msg_id = ar.msg_ids[0]
+        self.assertNotIn(msg_id, self.client.results)
+        self.assertNotIn(msg_id, self.client.metadata)
+
+    def test_non_owner(self):
+        self.minimum_engines(1)
+        
+        view = self.client[-1]
+        ar = view.apply_async(lambda : 1)
+        ar.owner = False
+        ar.get()
+        msg_id = ar.msg_ids[0]
+        self.assertIn(msg_id, self.client.results)
+        self.assertIn(msg_id, self.client.metadata)
+        ar2 = self.client.get_result(msg_id, owner=True)
+        self.assertIs(type(ar2), type(ar))
+        self.assertTrue(ar2.owner)
+        self.assertEqual(ar.get(), ar2.get())
+        ar2.get()
+        self.assertNotIn(msg_id, self.client.results)
+        self.assertNotIn(msg_id, self.client.metadata)
 
 
diff --git a/IPython/parallel/tests/test_client.py b/IPython/parallel/tests/test_client.py
--- a/IPython/parallel/tests/test_client.py
+++ b/IPython/parallel/tests/test_client.py
@@ -143,11 +143,12 @@ def test_get_result(self):
         ar = c[t].apply_async(wait, 1)
         # give the monitor time to notice the message
         time.sleep(.25)
-        ahr = self.client.get_result(ar.msg_ids[0])
-        self.assertTrue(isinstance(ahr, AsyncHubResult))
+        ahr = self.client.get_result(ar.msg_ids[0], owner=False)
+        self.assertIsInstance(ahr, AsyncHubResult)
         self.assertEqual(ahr.get(), ar.get())
         ar2 = self.client.get_result(ar.msg_ids[0])
-        self.assertFalse(isinstance(ar2, AsyncHubResult))
+        self.assertNotIsInstance(ar2, AsyncHubResult)
+        self.assertEqual(ahr.get(), ar2.get())
         c.close()
     
     def test_get_execute_result(self):
@@ -162,11 +163,12 @@ def test_get_execute_result(self):
         ar = c[t].execute("import time; time.sleep(1)", silent=False)
         # give the monitor time to notice the message
         time.sleep(.25)
-        ahr = self.client.get_result(ar.msg_ids[0])
-        self.assertTrue(isinstance(ahr, AsyncHubResult))
+        ahr = self.client.get_result(ar.msg_ids[0], owner=False)
+        self.assertIsInstance(ahr, AsyncHubResult)
         self.assertEqual(ahr.get().execute_result, ar.get().execute_result)
         ar2 = self.client.get_result(ar.msg_ids[0])
-        self.assertFalse(isinstance(ar2, AsyncHubResult))
+        self.assertNotIsInstance(ar2, AsyncHubResult)
+        self.assertEqual(ahr.get(), ar2.get())
         c.close()
     
     def test_ids_list(self):
@@ -450,6 +452,7 @@ def test_purge_local_results_outstanding(self):
         v = self.client[-1]
         ar = v.apply_async(lambda : 1)
         msg_id = ar.msg_ids[0]
+        ar.owner = False
         ar.get()
         self._wait_for_idle()
         ar2 = v.apply_async(time.sleep, 1)
diff --git a/IPython/parallel/tests/test_view.py b/IPython/parallel/tests/test_view.py
--- a/IPython/parallel/tests/test_view.py
+++ b/IPython/parallel/tests/test_view.py
@@ -142,11 +142,12 @@ def test_get_result(self):
         ar = v.apply_async(wait, 1)
         # give the monitor time to notice the message
         time.sleep(.25)
-        ahr = v2.get_result(ar.msg_ids[0])
-        self.assertTrue(isinstance(ahr, AsyncHubResult))
+        ahr = v2.get_result(ar.msg_ids[0], owner=False)
+        self.assertIsInstance(ahr, AsyncHubResult)
         self.assertEqual(ahr.get(), ar.get())
         ar2 = v2.get_result(ar.msg_ids[0])
-        self.assertFalse(isinstance(ar2, AsyncHubResult))
+        self.assertNotIsInstance(ar2, AsyncHubResult)
+        self.assertEqual(ahr.get(), ar2.get())
         c.spin()
         c.close()