lmJoy: A computational platform 

for the deep learning era 

Wei Ouyang 

Cell Profiling Group, Scilifelab, KTH, Stockholm 

Headed by Emma Lundberg 
Overview 

• Human Protein Atlas and HPA image classification competition 

• ANNA-PALM: Deep learning for super-resolution microscopy 

• lmJoy: A computational platform for the deep learning era 
The Human Protein Atlas 

THE HUMAN PROTEIN ATLAS •• • Mapping the human proteome using in-house generated E: MENU HELP NEWS proteome-wide collection of antibodies. 

e.g. RBM3, Insulin, C036 • Using large-scale immunostaining and high-resolution 

microscopy 

• Freely available database: www.proteinatlas.org 

HPA Cell Atlas 

Prof. Lundberg "Foreign Cell Sample" Classification 

PROJECT 

DISCOVERY 

Cl IIZEN SC IENCf flHi lN S WIT H YO U 

awrlt"l ,·wt•''>'•:.,,1·,· ·1!VI 

• >322,000 players on EVE online 

• 32,000,000 classifications (70 working years) 

Uh/en et al, A tissue based map of the human proteome, Science, 2015 

Thul et al, A subcellular map of the human proteome, Science, 2017 

Sullivan et al, D, Nature Biotechnol. 2018 
'!' Featured Prediction Competition 

Human Protein Atlas Image Classification $37,000 

Classify subcellular protein patterns in human cells 

• • Human Protein Atlas · 2,172 teams · 4 months ago 

Nucleus Protein of Interest Multi-label Prediction 

Mlcrotubules 

Classifier 

--+ 

Nucleoplasm 

Cytosol 

Plasma Membrane 

Nucleoli 

Mitochondria 

Golgi Apparatus 

Nuclear Bodies 

Nuclear Speckles 

Nucleoli Fibrillar C. 

Centrosome 

Cell Junctions 

Actin Filaments 

... 

Prize Money 

• >120,000 annotated images 

• $37,000 prize money 

• 3 months 

• >2,000 teams 

• >55,000 submissions 

• Top model: Densnet 121 + lovaza loss etc. 

• 20% higher score than previous record 

Deep learning + large image dataset+ crowdsourcing! Ouyang et. al, Nat Methods, 2019 
Deep learning accelerate

Super-resolution microsco
Artificial Neural Network Accelerated PALM/STORM 

raw images 300 frames 30,000 frames 

..--------------,-, Super-res Reconstruction Error 

~ L 1 + MS-SSIM 

ANNA-PALM(k) . Conditional GAN Error 

G CJ · [] -

,o.,m 

-r, 

' ' ' ' ANNA-PALM ' ' ', Stochastic Gradient Descent 

Artificial Neural Network Accelerated '---------------------------------------J PALM 

Betzig et al. Science 2006 Ouyang et. al, Nat Biotechnol, 2018 

Low-res Reconstruction Error 
ANNA-PALM: lOOx faster SR imaging 

Input Output Ground-truth 

nature 

biotechnology 

Deep learn ing for super-resolution imaging 

Sing le-cell li neage tracing and rranscnprnme profi ling 

Bra in organo1ds thrive in the bra in 

Ouyang et. al, Nat Biotechnol, 2018 
Web Application for ANNA-PALM 

• • 8 ANNA-PALM X + 

• No installation required ANNA-PALM artificial neural network accelerated localization microscopy -ft- HOME I! SHARELOC.XYZ • ABOUT LOGIN/SIGN UP 

< 

~ 

+ LOAD < SHARE • Not Scalable Flies 

• Data transmission + #smlm 2017-03-29-sp-tub-brb80-s1-c3-5-ds 

Apply filter 

.

click a name to set the value range: • Maintenance required! ,, 

0 RESET FILTER 'y FILTER EXPRESSION 

Adjust brightness 

• • Cost More options 

• ApplyLUT • Privacy, GDPR PixelSize{nm) Offset Mode 

20 min- max 

microtubule Type a tag and press enter ... 

pleaseusetagstolabelthissample.E.g. microtubule,hela,I 

0 Info Useful, but requires improvements! 
lmJoy: An open-source comput

platform for the deep learnin
Software for the deep learning ara 

• Large dataset, remote storage, online database (IDR, HPA) 

• Computationally hungry (GPU/TPU/NPU ... ) 

• Interactively inspect, annotate data and train model (human-in-the-loop) 

Inspection with Class Activation Maps 

classification CAM metric learning CAM Input image Model 1 Model 1 

Ouyang et. al, Nat Methods, 2019 
Opportunities 

• Deep learning libraries: Keras/Tensorflow, Pytorch 

• Mobile is the biggest platform 

o > 5 billion mobile devices 

• Cloud computing and remote data access 

o cloud services: storage, GPU, database, serverless, AutoML, Kubernetes 

o Free computation: Google Colab, MyBinder.org 

o Remote Storage: S3, NS, Zarr, Dask Array 

• Web standards 

o HTMLS standards, JS libraries, nodejs/npm 

o UI standards {e.g. material design) 

o Progressive Web App, offline support 

o WebAssembly {porting C/C++, C#, Rust, Python) 

o WebGPU {deep learning in the browser) 
--

tllmJoy 

• Hybrid computing modes 

(browser+ local + cloud) 

• Progressive Web App => Offline 

support, mobile support 

• Web Assembly/Web GPU=> 

browser based computing, security 

https://imjoy.io 

D 

-

I 

·

I 

<~-> ~ lmJoy browser app 

• • • 0 m,oy~ .,._ 

~ C O ~"' ""'°""' 1nc .... .., g 

~ lmJoybet.1 :t < em9ilnagaS!oun<I. 

iiiiiiiiiiiiiiiiiiiiimiiiiiiiiiiiiii• HPAv18 UMAP (model: deosenetl21, images; 67239) 

!!I HPA ·UMAP v 

.. Sklll•LeSIOll·ANlyzer V 

Plugin Engine 

Cloud 

servers 

0-0 

' 0 ✓ 

- Local cluster 

Local 

workstation 
WEB ASSEMBLY NumPy 

O sciPy 

~ pythonTM ]g 

Hardware and devices ~.~l~tkJ;m~ge Gxnet 

NumPy C,n 

matp© lib Tensorflow o,.J:! t r'. 

CI Keras PYT6 RCH Pandas 

·- lmJoy Web App Plugin Engine _/ ~  C ---------

~ CONDA 

lmJoy API < --W~e=b~So~ck~et~-> SocketlO Server 

' window ' 'web-worker ' 'web-python ' ' native-python ' 

three.js TensorFlow.js 

vega 

€3-itfri1⁄4J matp© lib 

Paper.js 

0 More backends 

~, 

Jupyter 

• ' -¥ 

<8 binder 

colab 

Minimal but powerfull 
tl lmJoy Key ideas 

• Each plugin import and export a set of service functions 

• Transparent, symmetrical Remote Procedure Calls {RPC) => across 

plugins/programming language/host {also see RPyC in Python) 

• Asynchronous execution => dynamic workflow composition 

(Inspired by Crossbar.io, Jailed.js, Tensorflow, Pytorch) 
ll I mJoy workflow composition: RPC + Async 

1 import asyncio 

2 from imjoy import api 

3 

4 class ImJoyPlugin (): 

5 async def setup( self ): 

self .pluginA = await api.getPlugin( 1 pasteur.fr/pluginA 1 ) 

self .pluginB = await api.getPlugin( 'kth.se/pluginB' ) 

self .pluginC = await api.getPlugin( 'pluginC' ) 

16 async def concurrent_workflow( self, x): 

17 promiseA = self .pluginA.process(x) 

18 promiseB = self .pluginB.process(x) 

19 resultA, resultB = await asyncio.gather(promiseA, promiseB) 

~0 L J result - await self .pluginC.process(resultA, resultB) 

return result 
tl lmJoy static vs dynamic workflow 

Fixed node/connection, file based io Self-organizing, dynamic node 

single workflow engine 

[ Define ] -[ Compile ] { ____ R_u_n __ ]

Tensorflow 1.x 

Pytorch Hard to debug! 

dynamic connection, data streaming 

Tensorflow 2.0 

Engine 1 
tllmJoy 

0 develop 

l!)lmJey --•---"-•••••• 

Predictions: 

.. .. ......, -·----- .. ~_, .. , ... _, ... -............. .. 

.. 

U ·-- ,_. ,._, _____ _ ___ 

,,.,..._. ,.,.., •. A __ -••• . ., ------tt> ~ . .. . _.. -.., ___ .... .. , ~ ,_,,_, 

~•--Of•••-

.. 

N 

------

__ -·.,..., --·--·· ___ .. ,-.-__ ..... .... _ .. ··-·_. ...,_ ........ __ ,, , .-. ... -~ .. , .. ;., __ ,,.,._._,, 

" -· ..... ........ __ ,_ ___ ,. .. ' ,, ____ , ... , .... 

- __ ,.;.·~~ " , .. _,_,_,..,_,, - ..... _ ·"--"'"'-·"'-"''· _,_, 

D 

@deploy 

8 share 

Ouse 

D 

6J https://imjoy.io/#/app?p=imjoy-team/example-plugins:Skin-Lesion-Analyzer 

Skin-Lesion-Analyzer ~ 

J~~LM O J~~t 

Plugin Repository 

Noise2self ff 

Static web app: high scalability & availability, rv zero cost 
tl lmJoy 

• Plugin repository 

demo plugins 

Image Annotator(annotation) 

HPA-UMAP(visualization) 

Skin-Lesion-Analyzer( classification) 

HPA-Classification(classification) 

··• interactive Plot(basic example) 

I! DeepBindScan(Genomics) 

··• Noise2Self(Denoising) 

··• ANNA-PALM(super-resolution) 

I! CARE(3D image denoising) 

··• DPNUnet(Segmentation) 

··• 1mageJ Demo(PylmageJ) 

(I! : requires plugin engine) 

lmJoy Plugln Repository 

~ ( >DEVELOPMENT ~ S Af l~Y I mJoy Plugin Repository 

Deep Learning Made Easy! 

# Plugins: 88 + 

~ (i..lll'f'-~"Mljllot1MUQ,na 1• )( na _ _,,..,_.,, )(-•y.1,-.,0.-~t•) (-av~l"lwQ111'11.u) ( ....,P'I",_,) 

c~ ......... , ) ( -wOIUl'.n )( ··.,.._· ). 

e 1rruoy • •-------i!ll~IIIIIII•< > ~ 

" , . - -­ ·-

. -, 

• CARE •9 - , ·---- ' Thl::splugin~.onstril!at.~ngu~ngCAAE • •• • ,; -.'"'· ,_ •._\~tt;-}~ 

. 

. , . 

. . ' . -. - . . ,. ~ . - ~ --) . . . .-: _•;;;-, • No1se2Vo1d ," . .,,}t"-., ' Tota ptug,n demOl\itl'ates denolSU'lg uwig Nocst12VOICI ~.,;,' ),::~:} ~1 . 

<> G < <> G < 

< 

-

_, -::: 

l!'J lmJoy 

. <> G < -~===-~---

t13904. 1763_F1_2 
Conclusion 

• HPA competition: large image database+ deep learning 

• ANNA-PALM: Deep learning accelerated SR imaging 

• lmJoy makes it easier to deploy deep learning models 

~ lmJoy o Progressive Web App, Web Assembly, HTMLS FOCUS I CORRESPONDENCE 

o Extendable Plugin Engines lmJoy: an open-source comp utational platform 

for the deep-learning era 

o Transparent Remote Procedure Calls 

o Async workflow composition 

o Static, scalable, high availability, almost zero cost 

lmJoy is now published in Nature Methods: https://rdcu.be/bYbGO 

(Deep Learning focused Issue) 

Work in progress: data viewers, plugin engine, useful plugins! 
Acknowledgements W @weioyang 

Thank all the Kaggle participants for their contributions. • 

SciL_ifoLab \ I' 

Lundberg Lab 

Ulrika Axelsson 

Anna Backstrom 

Jenny Fall 

Christian Gnann 

Martin Hjemare 

Diana Mahdessian 

Anna Martinez Casals 

Rutger Schutten 

Charlotte Stadler 

Devin Sullivan 

Peter Thul 

Casper Winsnes 

Hao Xu 

Lovisa Akesson 

Anthony Cesnik 

Human Protein Atlas 

• Mathias Uhlen 

• Fredrik Ponten 

• Adil Mardinoglu 

• Kalle von Feilitzen 

• Jan Mulder 

• Cecilia Lindskog 

• Burcu Ayoglu 

Collaborators 

• Manu Leonetti, CZ Biohub 

• Kathryn Lilley, Cambridge Univ 

• Oana Carja, CMU 

• Attila Szantner, MMOS 

• Andie Nordgren, CCP games 

• Bergur Finnbogasson, CCP games 

• EVE Online players 

0 CHAN ZUCKERBERG e CHAN BIOHUB ZUCK RBERG 

INITIATIVE 

Vetenskapsradet 
Acknowledgements W @weioyang 

Imaging & Modeling Unit @ lnstitut Pasteur 

• Christophe Zimmer 

Andrey Aristov, Benoit Lelandais, Christian Weber, Elena Rensen, 

Florian Muller, Jyotsana Parmar 

Mickael Lelek, Maxime Woringer, Xian Hao) 

• Cecile Leduc, Sandrine Etienne-Manneville 

• Sandrine Leveque-Fort, Nicolas Bourg 

• Orestis Faklaris, Julien Selles 

The IT service of Pasteur, Inception GPU lab 

Paris 

Pasteur University {"f I 

lnstitut Pasteur : PPU 110 _,, .. ,.,.,,N ___R 

FONDATI ON ~ 7 

RE~H ERCHE '#,., • 

MED ICALE ' 

...... -~ · \) •..,--C ·· ~ ' ' .. .. ,, .. I ,! , 

II,; ' J..' ""' ' J,, ..), . ..... -- ift_"'• Dim Malinf, .;_:! -·. · • .. . 

(Stephane Fournier, Thomas Menard, Jean-Baptiste Denis, Tru Huynh, et. al.) 
Thank you! 
Useful links for lmJoy W @weioyang 

Source code and documentations 

• documentation, API functions 

• main repo, pluqin-enqine 

Example Plugins 

• official pluqin repository, example pluqins 

Getting Help 

• lmaqe.sc Forum 

• lmJoy Slack 

Bug Report 

• lmJoy Issues 

Preprint on Nature Methods: https://rdcu.be/bYbGO