End-to-end learning of pharmacological assays from high-resolution microscopy images

Predicting the outcome of pharmacological assays based on high-resolution microscopy images of treated cells is a crucial task in drug discovery which tremendously increases discovery rates . However , end-to-end learning on these images with convolutional neural networks ( CNNs ) has not been ventured for this task because it has been considered infeasible and overly complex . On the largest available public dataset , we compare several state-of-the-art CNNs trained in an end-to-end fashion with models based on a cell-centric approach involving segmentation . We found that CNNs operating on full images containing hundreds of cells perform significantly better at assay prediction than networks operating on a single-cell level . Surprisingly , we could predict 29 % of the 209 pharmacological assays at high predictive performance ( AUC > 0.9 ) . We compared a novel CNN architecture called “ GapNet ” against four competing CNN architectures and found that it performs on par with the best methods and at the same time has the lowest training time . Our results demonstrate that end-to-end learning on high-resolution imaging data is not only possible but even outperforms cell-centric and segmentation-dependent approaches . Hence , the costly cell segmentation and feature extraction steps are not necessary , in fact they even hamper predictive performance . Our work further suggests that many pharmacological assays could be replaced by high-resolution microscopy imaging together with convolutional neural networks . 1 INTRODUCTION . High-resolution microscopy fluorescence imaging is an increasingly important biotechnology in the field of drug discovery ( Pepperkok and Ellenberg , 2006 ; Starkuviene and Pepperkok , 2007 ) . This imaging biotechnology captures morphological changes induced by chemical compounds on cell cultures in a very cost- and time-efficient way ( Yarrow et al. , 2003 ) and is viewed as remedy for the current drug discovery crisis ( Dorval et al. , 2018 ) . However , exploiting the wealth of information contained in those images for drug discovery is still a challenge ( Simm et al. , 2018 ) and it is important to lower entry barriers for the analysis of such data ( Scheeder et al. , 2018 ) . A typical step of conventional microscopy image analysis pipelines is to segment images into single cells and then extract cell-level feature vectors ( Carpenter et al. , 2006 ) . Individual steps in the pipeline of such approaches usually require optimization of the segmentation and feature extraction procedure to the specific cell culture or assay . This is a time-consuming process in which each step potentially introduces errors and uncertainty . Furthermore , parameters for each step are typically adjusted independently of subsequent steps ( Finkbeiner et al. , 2015 ; Sommer and Gerlich , 2013 ) . Conventional microscopy image analysis is centered on single cells and thus involves segmentation and feature extraction . Predicting pharmacological assays on the basis of high-resolution microscopy data has first been undertaken by Simm et al . ( 2018 ) using a cell-centric approach and has led to a tremendous increase of discovery rates , 250-fold and 60-fold , in two ongoing drug discovery projects . In principle , the authors replaced the chemical features of quantitative structure-activity relationship ( QSAR ) models with image-derived features . However , while the authors compare several machine learning methods , they use a traditional pipeline of first segmenting individual cells and deriving features from these . This might explain why all methods they compared performed equally well , as important information is already discarded during pre-processing . Using cell-centric approaches implies that the local neighborhood of cells , their spatial arrangement , and relation to other cells are lost . However , Bove et al . ( 2017 ) show that the neighborhood of cells and their relative orientation plays an important role for biological processes . This information is not accessible to a model which gets features based on single cells as input . Thus , learning a model in an end-to-end fashion directly from images would be highly desirable , not only because it removes the time-consuming and computationally demanding segmentation step , but also allows learning from the cell neighborhood and spatial arrangement of cells . Since 2012 , convolutional neural networks ( CNNs ) have been shown in several applications to outperform conventional methods in the field of image analysis especially where large datasets are available . CNNs can have a vastly better performance than expert systems at classifying images into thousands of categories ( Krizhevsky et al. , 2012 ; Simonyan and Zisserman , 2014 ; He et al. , 2015 ) , recognizing traffic signs ( Ciresan et al. , 2011 ) and even pixel-accurate segmentation of images ( Long et al. , 2015 ) . As CNNs learn features from images automatically they can easily be adapted to other domains and indeed have been applied to cellular image data , for example , to segment cells achieving higher accuracy than human experts ( Ronneberger et al. , 2015 ) . Therefore , CNNs are a promising method for extracting biological knowledge from high-resolution imaging data for the purpose of drug design . However , applying CNNs to high-resolution microscopy imaging data in the field of drug design poses unique challenges . State-of-the-art CNN architectures are designed for and evaluated on benchmark datasets such as ImageNet ( Everingham et al. , 2010 ; Lin et al. , 2014 ; Russakovsky et al. , 2015 ) in which images are of a much lower resolution than typical microscopy images . This poses a problem as increased image resolution immediately results in vastly increased memory consumption , especially for very deep architectures , making training infeasible with full resolution microscopy images while scaling or cropping such images results in loss of information . Another challenge arises from the fact that labels for microscopy images are often noisy , as typically whole images are assigned a label by experts but not all individual cells conform to this label . In Kraus et al . ( 2016 ) the authors combine methods from Multiple Instance Learning and convolutional neural networks to alleviate this problem and in the process learn to focus on correctly labeled cells . Overall , the main challenges of high-resolution image analysis of cells are handling the high resolution of the images and that typically the whole image rather than a single cell is labeled . We use the largest public dataset , Cell Painting ( Bray et al. , 2017 ) , which consists of high-resolution fluorescence microscopy images of cells treated with chemical compounds , to benchmark and compare convolutional neural networks against each other and against the best feature-based method . This dataset comprises 919,265 five-channel microscopy images across 30,610 tested compounds as well as single-cell features extracted using a CellProfiler ( Carpenter et al. , 2006 ) pipeline . We augmented this dataset with drug activity data for 10,574 compounds integrated from ChEMBL ( Gaulton et al. , 2017 ) . However , we do not restrict activity information from ChEMBL to specific assays as that would reduce the amount of data available drastically . Therefore , the resulting labels must be viewed as noisy and learning meaningful patterns from this dataset is no easy task . We introduce a novel network architecture that is able to cope with the characteristics of typical microscopy images , GapNet . Our architecture extracts features from full resolution images such that there is no need for scaling or cropping the input images . It then combines features from different levels of abstraction and spatial resolution before feeding the resulting features into a fullyconnected classification network . We combine the features in such a way that the network can handle arbitrarily large input images . In Section 3 , we provide details on the datasets , we describe competing state-of-the-art methods previously used for similar data and the evaluation criteria , and we introduce our novel network architecture . Finally , in section 4 we present and discuss results . 2 DATASET . To assess our method we use a dataset released by Bray et al . ( 2017 ) which we refer to as Cellpainting Dataset . This dataset contains 919,265 five-channel microscopy images ( using the U2OS cell-line ) across 30,610 tested compounds as well as single-cell features extracted using a CellProfiler ( Carpenter et al. , 2006 ) pipeline . For our comparison we used the most recent version of the pre-computed features , where for each image 1,783 features were extracted . The fact that the cells in each image were treated with a specific chemical compound allows to automatically obtain labels for this dataset without the need of labeling by a human expert . We identify each chemical compound in the large bioactivity database ChEMBL ( Gaulton et al. , 2017 ) . This database provides outcomes of biochemical tests , called assays , such as a compound ’ s ability to inhibit a certain receptor or whether a compound exhibits a certain toxic effect . In this way , we obtained labels for this dataset as drug activity data for 10,574 compounds across 209 assays . Consequently , we created a large dataset of high-resolution images together with chemical activity data . Concretely , our computational pipeline for labeling the images comprises the following steps : We first convert the SMILES representation of compounds into the InChiKey ( Heller et al. , 2013 ) format . We then query the ChEMBL database for compounds matching these InChiKeys resulting in 11,585 hits . For these compounds we extract two values from the database , namely pChEMBL , a numerical value on log scale indicating bioactivity , and activity comment , where the researcher or lab technician creating the entry on ChEMBL marked a compound as either active or inactive . We obtain the first part of our label matrix by extracting all measurements with a pChEMBL value between 4 and 10 for IC50 ( inhibitory effect ) or EC50 ( stimulatory effect ) . However , due to the high amount of noise in these labels , we aim at binary prediction tasks ( active/inactive ) and therefore threshold this matrix similar to other works on assay or target prediction ( Mayr et al. , 2018 ) . We apply not only one but three increasing thresholds , namely 5.5 , 6.5 and 7.5 . As the pChEMBL value is on a log scale these thresholds represent increasingly confident activity indicators . Applying all three thresholds allows us to obtain more labeling information , hence we concatenate the three resulting matrices along the assay dimension . This means that an assay can occur multiple times in the final label matrix but at different thresholds . The second part we obtain by extracting all compounds with a valid activity comment . We only allow a defined set of comments ( such as `` active '' or `` inactive '' with slight variations in spelling and casing ) . We combine the results of both the thresholded pChEMBL values as well as the activity comment along the assay dimension and filter only for those assays , or in case of the first part assay/threshold combinations , where at least 10 active and 10 inactive compounds are present and remove compounds without any measurements in the remaining assays . The final label matrix consists of 10,574 rows corresponding to compounds , 209 columns corresponding to bioactivity assays with 0.87 % positive labels ( active ) , 1.64 % negative labels ( inactive ) and 97.49 % missing labels ( NA ) . Upon publication we will release our label matrix containing bioactivity data for the Cellpainting Dataset , for which images and pre-computed features are publicly available . From the Cellpainting Assay dataset we extract images corresponding to compounds for which we have activity information in our label matrix . As the dataset contains multiple screens per compound we have several images per row in the label matrix . Furthermore , each screen is comprised of six adjacent images , called views , with a resolution of 692×520 and 5 channels , each channel corresponding to a stain used for the microscopy screen . Since individual views already contain a large number of cells we do not combine these images to obtain one large image per screen but rather use each view image individually for training and only combine the network outputs by averaging predictions . Figure 1 shows examples of images from the assay `` Gametocytocidal compounds screen '' labeled as ( a ) active and ( b ) inactive ( for illustrative purposes , each image is a full screen image comprised of six views ) . The final dataset consists of 284,035 view images which we split into training- , validation- and test set making sure that multiple images from the same sample are in the same fold . We used 70 % of compounds for training ( corresponding to 198,609 images ) , 10 % as a validation set ( 28,632 images ) and the remaining 20 % ( 56,794 images ) were held out for testing the final performance . Due to the sparseness of the label matrix , the majority of output units for a given sample should not receive an error signal . Therefore , the loss for all output units for unlabeled assays for a given sample are masked by multiplying it with zero before performing back-propagation to update the parameters of the network during training .

Summary: The authors explore the possibility of using an end-to-end approach for predicting pharmacological assay outcome using fluorescence microscopy images from the public Cell Painting dataset. In my view, the primary contributions are the following: an interesting and relatively new application (predicting assay outcomes), enriching the CellPainting dataset with drug activity data, and a comparison of several relevant methods and architectures. The technical novelty is weak, and although the authors demonstrate that end-to-end holistic approaches outperform previous segmentation-and-feature-extraction approaches, this result is not surprising and has been previously reported in closely related contexts.