ebbwdan/omero_scripts — pull request #1 by will-moore (2014-08-19 12:49:58+00:00)
Title: Use getTiles() instead of getTile()

Hi Dan, here's one way of making Slide_Scanner_Images_From_ROIs a little faster.

If you use one call to pixels.getTiles() instead of many calls to getTile() then OMERO only has to open ONE rawPixelsStore for all the tiles instead of one per tile.

The caveat is that you have to know the exact sequence in which the tiles are going to be requested.
I have taken that sequence from util.tiles.py, but this might change (especially since it's a different order from that used in conn.createImageFromNumpySeq().

I tested on my local laptop server with an ROI of 15410 x 10833, 16 x 11 x 3 channels = 528 tiles.
Using your original script it took 560 secs.
Using my adapted script took 315 secs.
I also tried it with a fake tile of numpy data (same as Josh's example) and this took 227 secs, which shows that a fair bit of the time is taken with setTile().

Just looking at the Blitz logs for each of these examples.
Original script: Each tile opens & closes rawPixelsStore. For ONE tile:
```
2014-08-19 11:32:33,707 INFO [ omero.cmd.SessionI] (l.Server-5) Added servant to adapter: e104741b-035f-4ab2-8365-2fc0792987a6/a1b2c59a-5271-4911-b406-ec0660548c78omero.api.RawPixelsStore(omero.api._RawPixelsStoreTie@e9def82d)
2014-08-19 11:32:33,709 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Meth:	interface ome.api.RawPixelsStore.setPixelsId
2014-08-19 11:32:33,709 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Args:	[13, true]
2014-08-19 11:32:33,710 INFO [ ome.security.basic.EventHandler] (l.Server-1) Auth:	user=3,group=5,event=null(User),sess=e104741b-035f-4ab2-8365-2fc0792987a6
2014-08-19 11:32:33,712 INFO [ ome.io.nio.FilePathResolver] (l.Server-1) Metadata only file, resulting path: /OMERO/ManagedRepository/will_3/2014-07/24/11-26-03.523/16_Elastic artery.svs
2014-08-19 11:32:33,735 DEBUG [ loci.formats.Memoizer] (l.Server-1) loaded memo file: /OMERO/BioFormatsCache/OMERO/ManagedRepository/will_3/2014-07/24/11-26-03.523/.16_Elastic artery.svs.bfmemo (611559 bytes)
2014-08-19 11:32:33,735 DEBUG [ loci.formats.Memoizer] (l.Server-1) start[1408444353713] time[22] tag[loci.formats.Memoizer.loadMemo]
2014-08-19 11:32:33,735 DEBUG [ loci.formats.Memoizer] (l.Server-1) start[1408444353712] time[22] tag[loci.formats.Memoizer.setId]
2014-08-19 11:32:33,735 INFO [ ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /OMERO/ManagedRepository/will_3/2014-07/24/11-26-03.523/16_Elastic artery.svs Series: 0
2014-08-19 11:32:33,736 INFO [ org.perf4j.TimingLogger] (l.Server-1) start[1408444353709] time[27] tag[omero.call.success.ome.services.RawPixelsBean.setPixelsId]
2014-08-19 11:32:33,736 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Rslt:	null
2014-08-19 11:32:33,737 INFO [ ome.services.util.ServiceHandler] (.Server-10) Meth:	interface ome.api.RawPixelsStore.getTile
2014-08-19 11:32:33,737 INFO [ ome.services.util.ServiceHandler] (.Server-10) Args:	[0, 2, 0, 15681, 7649, 1024, 1024]
2014-08-19 11:32:33,738 INFO [ ome.security.basic.EventHandler] (.Server-10) Auth:	user=3,group=5,event=null(User),sess=e104741b-035f-4ab2-8365-2fc0792987a6
2014-08-19 11:32:34,027 INFO [ org.perf4j.TimingLogger] (.Server-10) start[1408444353737] time[290] tag[omero.call.success.ome.services.RawPixelsBean.getTile]
2014-08-19 11:32:34,028 INFO [ ome.services.util.ServiceHandler] (.Server-10) Rslt:	[-46, -47, -27, ... 1 more]
2014-08-19 11:32:34,179 INFO [ ome.services.util.ServiceHandler] (l.Server-7) Meth:	interface ome.api.StatefulServiceInterface.close
2014-08-19 11:32:34,179 INFO [ ome.services.util.ServiceHandler] (l.Server-7) Args:	()
2014-08-19 11:32:34,189 INFO [ ome.security.basic.EventHandler] (l.Server-7) Auth:	user=3,group=5,event=107770(User),sess=e104741b-035f-4ab2-8365-2fc0792987a6
2014-08-19 11:32:34,190 INFO [ org.perf4j.TimingLogger] (l.Server-7) start[1408444354179] time[11] tag[omero.call.success.ome.services.RawPixelsBean.close]
2014-08-19 11:32:34,190 INFO [ ome.services.util.ServiceHandler] (l.Server-7) Rslt:	null
2014-08-19 11:32:34,190 INFO [ ome.services.util.ServiceHandler] (l.Server-7) Cleanup: ome.services.RawPixelsBean@727008ec
2014-08-19 11:32:34,191 INFO [ omero.cmd.SessionI] (l.Server-7) Unregistered servant:e104741b-035f-4ab2-8365-2fc0792987a6/a1b2c59a-5271-4911-b406-ec0660548c78omero.api.RawPixelsStore(omero.api._RawPixelsStoreTie@e9def82d)
2014-08-19 11:32:34,344 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Meth:	interface ome.api.RawPixelsStore.setTile
2014-08-19 11:32:34,344 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Args:	[[-46, -47, -27, ... 1 more], 0, 2, 0, 7168, 1024, 1024, 1024]
2014-08-19 11:32:34,345 INFO [ ome.security.basic.EventHandler] (l.Server-5) Auth:	user=3,group=5,event=null(User),sess=e104741b-035f-4ab2-8365-2fc0792987a6
2014-08-19 11:32:34,565 INFO [ org.perf4j.TimingLogger] (l.Server-5) start[1408444354344] time[220] tag[omero.call.success.ome.services.RawPixelsBean.setTile]
2014-08-19 11:32:34,565 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Rslt:	null
```

With my updated script, for each tile we only have getTile() and setTile()
```
2014-08-19 13:18:55,418 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Meth:	interface ome.api.RawPixelsStore.getTile
2014-08-19 13:18:55,418 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Args:	[0, 0, 0, 8192, 5120, 1024, 1024]
2014-08-19 13:18:55,419 INFO [ ome.security.basic.EventHandler] (l.Server-5) Auth:	user=3,group=5,event=null(User),sess=83b50403-b811-4c26-9102-e1b1da7d1b5f
2014-08-19 13:18:55,596 INFO [ org.perf4j.TimingLogger] (l.Server-5) start[1408450735418] time[177] tag[omero.call.success.ome.services.RawPixelsBean.getTile]
2014-08-19 13:18:55,596 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Rslt:	[-20, -20, -20, ... 1 more]
2014-08-19 13:18:55,775 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Meth:	interface ome.api.RawPixelsStore.setTile
2014-08-19 13:18:55,775 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Args:	[[-20, -20, -20, ... 1 more], 0, 0, 0, 8192, 5120, 1024, 1024]
2014-08-19 13:18:55,776 INFO [ ome.security.basic.EventHandler] (l.Server-1) Auth:	user=3,group=5,event=null(User),sess=83b50403-b811-4c26-9102-e1b1da7d1b5f
2014-08-19 13:18:55,917 INFO [ org.perf4j.TimingLogger] (l.Server-1) start[1408450735775] time[142] tag[omero.call.success.ome.services.RawPixelsBean.setTile]
2014-08-19 13:18:55,918 INFO [ ome.services.util.ServiceHandler] (l.Server-1) Rslt:	null
```
And just for comparison, using the fake tile we only have setTile()
```
2014-08-19 11:50:39,927 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Meth:	interface ome.api.RawPixelsStore.setTile
2014-08-19 11:50:39,927 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Args:	[[0, 0, 0, ... 1 more], 0, 0, 0, 4096, 3072, 1024, 1024]
2014-08-19 11:50:39,928 INFO [ ome.security.basic.EventHandler] (l.Server-5) Auth:	user=3,group=5,event=null(User),sess=6a93b0bf-9f33-4b97-8200-d13b639dffc0
2014-08-19 11:50:40,013 INFO [ org.perf4j.TimingLogger] (l.Server-5) start[1408445439927] time[85] tag[omero.call.success.ome.services.RawPixelsBean.setTile]
2014-08-19 11:50:40,013 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Rslt:	null
```