<figure><img src="image_1.png" /></figure>

HAL
open science

# Transfer Learning for Predicting Gene Regulatory Effects of Chemicals

Bahattin Can Maral, Mehmet Tan

► To cite this version:

Bahattin Can Maral, Mehmet Tan. Transfer Learning for Predicting Gene Regulatory Effects of Chemicals. 18th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2022, Hersonissos, Greece. pp.414-425, 10.1007/978-3-031-08337-2_34. hal-04668672

HAL Id: hal-04668672

https://inria.hal.science/hal-04668672v1

Submitted on 7 Aug 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Transfer Learning for Predicting Gene Regulatory
Effects of Chemicals

Bahattin Can Maral[0000-0002-1552-1592] and Mehmet Tan[0000-0002-1741-0570]

TOBB University of Economics and Technology, Sogutozu, Ankara, Turkey
[bahattincanmaral,mtan]@etu.edu.tr

**Abstract.** Among the recent developments in bioinformatics and chemogenomics, various deep learning methods have been the most prevalent [4, 5, 9]. This resulted in an over-saturation of powerful models that easily pushed the limits of existing datasets. Subsequently, many novel advancements have been done with improvements to the datasets. Amidst these advancements, researchers of Deep Compound Profiler (DeepCOP) [10] set themselves apart with a novel method of introducing new features whilst keeping the deep learning model relatively basic. In this study, we propose to take this novel method one step further by applying transfer learning between cell lines. In order to better evaluate the benefits of transfer learning, we've introduced 2 drug-based data splits. The transfer learning method, as its core, utilizes the learned knowledge of "source" cell lines to give a head start to "target" cell lines. Taking advantage of prior knowledge from source cell lines not only boosts existing compounds' effect prediction, but it can also be used as a premonition for the compounds' (explicit to the source), effects on target cell lines before they are tested in real life. Our experiments showed improvements up to 22.81% improvement on area under ROC curve (AUC) on the split closest to a wet lab experiment.

**Keywords:** Transfer learning · Chemogenomics · Domain adaptation.

# 1 Introduction

Chemogenomics [1] studies chemical compounds and their genomic/proteomic effects on biological systems.

Since machine learning, chemogenomics has been advancing rapidly, the ability to simulate chemical effects on different cell lines with almost no cost also helped the drug-discovery process by identifying potential side effects of the candidate drugs early.

DeepCOP is a recently proposed deep learning model that is trained on the LINCS L1000 [8] transcriptomics dataset to predict differential gene expression effects of chemicals. In the study, researchers train two deep learning models in the form of a multilayer perceptron (MLP) for each selected cell line.

The study distinguishes itself with the heavy focus on feature generation and uncomplicated model structure.
In this study, we aim to take the work done by DeepCOP researchers one step further by introducing transfer learning to utilize the knowledge of different cell lines.

In machine learning, transfer learning is defined as the use of previously acquired knowledge gained while solving one problem on a different but related problem. For this study, different problems correspond to different cell lines. Utilizing the data from other cell lines also comes with the advantage of training a more general model due to there being experiments with chemical compounds unique to some cell lines. In addition to transfer learning, we have introduced 2 splitting methods that we think better represents the high-throughput screening process and 17 previously discarded cell lines.

This paper includes 3 key sections: Materials and methods, discussion, and conclusion. In the first of which, we briefly explain preparation of the dataset, clarify the key differences between the splitting methods, and give average improvements for each method. The discussion section mainly focuses on splitting complications, what can be done to counter them, and what we did in this study. In the conclusion section, we wrap up the paper, and give a short summary for the planned studies moving forward.

# 2 Materials and methods

We aim to prove the usefulness of transfer learning for the prediction of chemical induced differential gene expression. To produce comparable results to DeepCOP, the data production and preparation steps have been replicated, and the novel ideas repeated with the original methodology.

## 2.1 Datasets

L1000 dataset provides the data in the form of levels. In this study, we utilize the Level 5 dataset. Level 5 data comprise the gene profiles of 978 landmark genes' drug-gene interaction experiments on distinct cell lines. To get to the Level 5 of the L1000 dataset, the same method of genex measurement, namely flow cytometry, is used throughout levels 1 to 5. At Level 5 these values are reformed into standardized z-scores. Further information about this dataset can be found in [8].

For data preparation, we've closely followed DeepCOP; with the only difference being the removal of the 10% threshold, for the sake of space limitations.

In DeepCOP, from the processed dataset, researchers selected the top six cell lines that had the largest number of drug-gene interaction experiments i.e., VCAP, A549, A375, PC3, MCF7, and HT29. For this study, we expanded this selection to every cell line that had more than 10 samples, which resulted in 17 additional cell lines compared to [10]. These additional cell lines are as follows: BT20, HA1E, HCC515, HEK293T, HEPG2, HL60, HS578T, HUH7, JURKAT, MCF10A, MDAMB231, NKDBA, NOMO1, SKBR3, THP1, U266, and U937 where details are given in Table 1.
<table><caption>**Table 1.** The details of the cell line id's that are included in this study.</caption><thead><tr><th>Cell Line</th><th>Primary Site</th><th>Sample Type</th><th>Subtype</th></tr></thead><tbody><tr><td>A375</td><td>skin</td><td>tumor</td><td>malignant melanoma</td></tr><tr><td>A549</td><td>lung</td><td>tumor</td><td>non small cell lung cancer<br>carcinoma</td></tr><tr><td>BT20</td><td>breast</td><td>tumor</td><td>carcinoma</td></tr><tr><td>HA1E</td><td>kidney</td><td>normal</td><td>normal kidney</td></tr><tr><td>HCC515</td><td>lung</td><td>normal</td><td>carcinoma</td></tr><tr><td>HEK293T</td><td>kidney</td><td>normal</td><td>embryonal kidney</td></tr><tr><td>HEPG2</td><td>liver</td><td>tumor</td><td>hepatocellular carcinoma</td></tr><tr><td>HL60</td><td>haematopoietic,<br>lymphoid tissue</td><td>tumor</td><td>acute myelogenous leukemia,<br>promyelocytic</td></tr><tr><td>HS578T</td><td>breast</td><td>tumor</td><td>carcinoma</td></tr><tr><td>HT29</td><td>large intestine</td><td>tumor</td><td>colorectal adenocarcinoma</td></tr><tr><td>HUH7</td><td>liver</td><td>tumor</td><td>hepatocellular carcinoma</td></tr><tr><td>JURKAT</td><td>haematopoietic,<br>lymphoid tissue</td><td>tumor</td><td>acute lymphoblastic leukemia,<br>T-cell</td></tr><tr><td>MCF10A</td><td>breast</td><td>normal</td><td>epithelial</td></tr><tr><td>MCF7</td><td>breast</td><td>tumor</td><td>adenocarcinoma</td></tr><tr><td>MDAMB231</td><td>breast</td><td>tumor</td><td>adenocarcinoma</td></tr><tr><td>NKDBA</td><td>kidney</td><td>normal</td><td>kidney epithelial</td></tr><tr><td>NOMO1</td><td>haematopoietic,<br>lymphoid tissue</td><td>tumor</td><td>acute myeloid leukemia</td></tr><tr><td>PC3</td><td>prostate</td><td>tumor</td><td>adenocarcinoma</td></tr><tr><td>SKBR3</td><td>breast</td><td>tumor</td><td>adenocarcinoma</td></tr><tr><td>THP1</td><td>haematopoietic,<br>lymphoid tissue</td><td>tumor</td><td>acute myelogenous leukemia,<br>monocytic</td></tr><tr><td>U266</td><td>blood</td><td>tumor</td><td>myeloman, haematopoietic,<br>lymphoid</td></tr><tr><td>U937</td><td>haematopoietic,<br>lymphoid tissue</td><td>tumor</td><td>lymphoma, B-cell, non-hodgkin's,<br>histiocytic</td></tr><tr><td>VCAP</td><td>prostate</td><td>tumor</td><td>carcinoma</td></tr></tbody></table>

## 2.2 Feature generation

Similar to the well-known Extended-connectivity fingerprints (ECFPs) or Functional-class fingerprints (FCFPs), Morgan fingerprints are used to represent chemical structures as binary arrays. From the first time, it was introduced[3], to the modernized implementation in [7], Morgan fingerprints have been one of the most prevalent fingerprinting methods to this day. In this study, Morgan descriptors were generated using the RDKIT Open-source cheminformatics toolkit[6] to correlate the chemical structure to gene perturbation. The descriptors on the conventional SMILES form were computed with a radius of 2 to generate a one-hot vector of 2048 features for 19 811 compounds.

While there are no widely acknowledged techniques for describing a gene, the gene ontology (GO) consortium [2], with 40 thousand GO terms and over 200
000 qualitative annotations for Homo sapiens, is the most prominent. DeepCOP proposes a novel way to utilize the GO descriptors along with the 978 landmark genes. Using the OntologyX R package, GO terms that correlated with at least 3 landmark genes were extracted. Then each gene was described using the appropriate GO terms. One-hot encoding of these GO descriptors resulted in a binary array of size 1107 that can be used as features alongside the Morgan fingerprints.

## 2.3 Data splitting

In DeepCOP, researchers have split the aforementioned dataset into 10 randomly selected folds to perform 10-fold cross-validation for each cell line. This approach ensures every compound-gene combination is unique per fold. However, it does not accommodate the fact that each drug is repeated by the number of landmark genes. Conversely, each gene is repeated by the number of compounds. Therefore, a model trained on randomly split 9 folds, has already encountered the input elements (compound and gene) for the remaining fold, separately. This leads to complications that may cause over-fitting, for the reasons that we further addressed in the discussion section. For the sake of the completeness of this study, we have also repeated our experiments in this format which will be called “random-split” moving forward.

Cold-Drug splitting method divides the compound data into 10 folds, then populates each fold with the gene data. This ensures the drugs in the test fold are brand new for the trained models. We named this method of splitting “cold-drug-split”. For transfer learning, the source model is trained on a 95%-5% random-split. Then we train the target cell line data on the cold-drug-split. We consider this splitting method to be the closest to real-life high-throughput testing.

Similar to cold-drug-split, we introduced Transfer-Drug splitting to counteract contamination of the test data. The main difference between the two splits is that this splitting method also eliminates the contamination between cell lines while transfer learning. On the last step of transfer learning on cold-drug-split, the compounds used in the training of the source model are removed from the test fold. This guarantees that the compounds in the test data are never-before-seen for the model. Going forward, this splitting method will be referred to as “transfer-drug-split”.

## 2.4 Experiments

This study aims to justify the value of utilizing previously tested knowledge of cell lines in novel test domains. To be able to make a direct comparison with DeepCOP results, we have repeated the original experiments both with a random split and a transfer-drug split as a baseline. During these experiments, we also discovered that similar scores could be achieved with much smaller neural networks from the original study. The results of the experiments listed below have been achieved with a neural network that reduces the neuron count of the original 2 hidden layers to 400 (from 3155 of DeepCOP), while the rest of the network structure remains unchanged.
Based on the assumption that learned knowledge from cell lines could ben-
efit others, we’ve opted to use network-based parameter sharing as our transfer
learning method. Parameter sharing methods, transfer their learned knowledge
between problems by keeping a part of or the whole network trained on the
donor/source problem. Continuing the training on the receiver/target problem
on the transferred network by either modifying the received weights or adding
brand-new layers to train.

In this study, we trained a source model for every single cell line with 95% train and 5% validation split data. Then we saved the finalized weights of the source model and used them as the initial weights for training the target cell line model. We repeated this process for every cell line pair. Network structure and parameters were kept unchanged throughout experiments to ensure com- parability. We selected the original six cell lines of the DeepCOP study as our source models because there was a huge falloff in the sample count after these six. The more knowledge we can acquire in the source problem makes for a stronger baseline for the target.

As our first objective, we trained target cell lines on the same random-split
with DeepCOP. Evaluating our model on random-split resulted in average im-
provements of 4.51% for up-regulated and 4.52% for down-regulated models'
AUC scores. Among up-regulated models, the maximum recorded betterment
was on the NKDBA cell line when MCF7 was used as a source; which resulted
in 15.73% improvement of the AUC score. As for down-regulated models, the
maximum improvement was on the HL60 cell line with A375 as the source; that
resulted in an added 12.73% for the AUC score.

A detailed view of the random-split experiments can be seen on the Table 2,or up and down-regulated model results. Each row represents the different target
cell lines, and each column corresponds to different sources, while the “No-TL”
(no transfer learning) column represents the results of the models when the
model is trained on the same split, without a source model, on target cell line
data. When the data is viewed as a whole, a clear trend emerges: The least
amount of improvements are seen on the experiments that are trying to improve
upon the cell lines we’ve selected as the source. This directly correlates to the
number of samples each cell line possesses. For the remaining 17 target cell
lines, the average gains for the AUC scores are 6.09% and 6.04% for up and
down regulation models. While the scores are impressive, the random-split is the
least realistic of the three. Transcriptomics datasets, like L1000, are subsets of
gene perturbation measurements of wet-lab experiments. In these experiments, a
chemical compound is tested on a cell line and the perturbations of the genes are
measured simultaneously. Therefore, predicting a gene’s perturbation by utilizing
other gene’s responses, is only useful in a situation where that specific gene
perturbation value is lost. Which isn’t useful for simulating high-throughput
screening.

Apart from being unrealistic, this became a visible issue while we were try-
ing to improve DeepCOP’s scores on isolation with network optimization, for a
stronger baseline. Our experiments showed minimal change in evaluation, even
<table><caption>**Table 2.** AUC scores of regulated genes on random-drug-split 10-fold data, rows and columns indicate target and source cell lines. The best results in a row are indicated in bold.</caption><thead><tr><th rowspan="2">Target Cell ↓</th><th colspan="7">Source Cell</th></tr><tr><th>No-TL</th><th>A375</th><th>A549</th><th>HT29</th><th>MCF7</th><th>PC3</th><th>VCAP</th></tr></thead><tbody><tr><td>A375</td><td><strong>0.8199</strong></td><td></td><td>0.8189</td><td>0.8190</td><td>0.8141</td><td>0.8167</td><td>0.8105</td></tr><tr><td>A549</td><td><strong>0.8254</strong></td><td>0.8243</td><td></td><td>0.8228</td><td>0.8246</td><td>0.8244</td><td>0.8222</td></tr><tr><td>BT20</td><td>0.7698</td><td>0.7975</td><td><strong>0.8015</strong></td><td>0.7994</td><td>0.8011</td><td>0.7987</td><td>0.7944</td></tr><tr><td>HA1E</td><td>0.8281</td><td><strong>0.8298</strong></td><td>0.8293</td><td>0.8281</td><td>0.8260</td><td>0.8297</td><td>0.8251</td></tr><tr><td>HCC515</td><td>0.8397</td><td>0.8405</td><td>0.8400</td><td>0.8403</td><td>0.8384</td><td><strong>0.8413</strong></td><td>0.8392</td></tr><tr><td>HEK293T</td><td>0.8011</td><td>0.8549</td><td>0.8533</td><td>0.8431</td><td>0.8503</td><td><strong>0.8576</strong></td><td>0.8495</td></tr><tr><td>HEPG2</td><td>0.8216</td><td>0.8548</td><td>0.8572</td><td><strong>0.8584</strong></td><td>0.8534</td><td>0.8517</td><td>0.8412</td></tr><tr><td>HL60</td><td>0.8427</td><td><strong>0.8964</strong></td><td>0.8836</td><td>0.8928</td><td>0.8872</td><td>0.8852</td><td>0.8835</td></tr><tr><td>HS578T</td><td>0.7556</td><td>0.7901</td><td><strong>0.7976</strong></td><td>0.7901</td><td>0.7909</td><td>0.7932</td><td>0.7859</td></tr><tr><td>HT29</td><td>0.7904</td><td><strong>0.7966</strong></td><td>0.7902</td><td></td><td>0.7875</td><td>0.7899</td><td>0.7822</td></tr><tr><td>HUH7</td><td>0.7505</td><td>0.8038</td><td>0.7998</td><td><strong>0.8072</strong></td><td>0.8048</td><td>0.8032</td><td>0.7987</td></tr><tr><td>JURKAT</td><td>0.8071</td><td>0.8579</td><td>0.8493</td><td><strong>0.8602</strong></td><td>0.8029</td><td>0.8536</td><td>0.8452</td></tr><tr><td>MCF10A</td><td>0.7733</td><td><strong>0.8054</strong></td><td>0.8002</td><td>0.8017</td><td>0.8013</td><td>0.7988</td><td>0.7928</td></tr><tr><td>MCF7</td><td>0.8313</td><td>0.8291</td><td>0.8315</td><td>0.8293</td><td></td><td><strong>0.8327</strong></td><td>0.8295</td></tr><tr><td>MDAMB231</td><td>0.7662</td><td>0.8207</td><td>0.8234</td><td>0.8214</td><td>0.8212</td><td><strong>0.8258</strong></td><td>0.8176</td></tr><tr><td>NKDBA</td><td>0.6833</td><td>0.7876</td><td>0.7891</td><td>0.7864</td><td><strong>0.7908</strong></td><td>0.7891</td><td>0.7878</td></tr><tr><td>NOMO1</td><td>0.7828</td><td>0.8013</td><td><strong>0.8278</strong></td><td>0.7485</td><td>0.7840</td><td>0.7816</td><td>0.7731</td></tr><tr><td>PC3</td><td><strong>0.8327</strong></td><td>0.8253</td><td>0.8276</td><td>0.8248</td><td>0.8279</td><td></td><td>0.8268</td></tr><tr><td>SKBR3</td><td>0.7655</td><td>0.7868</td><td>0.7943</td><td>0.7918</td><td><strong>0.7991</strong></td><td>0.7933</td><td>0.7855</td></tr><tr><td>THP1</td><td>0.7296</td><td>0.7492</td><td>0.7986</td><td>0.7805</td><td>0.7887</td><td>0.7883</td><td><strong>0.8130</strong></td></tr><tr><td>U266</td><td>0.7898</td><td>0.8146</td><td>0.8288</td><td><strong>0.8327</strong></td><td>0.7948</td><td>0.8307</td><td>0.8308</td></tr><tr><td>U937</td><td>0.7572</td><td>0.8023</td><td>0.8019</td><td><strong>0.8085</strong></td><td>0.7721</td><td>0.7993</td><td>0.7898</td></tr><tr><td>VCAP</td><td><strong>0.8471</strong></td><td>0.8457</td><td>0.8458</td><td>0.8450</td><td>0.8469</td><td>0.8466</td><td></td></tr></tbody></table>

<table><tbody><tr><td>A375</td><td>0.8239</td><td></td><td>0.8244</td><td><strong>0.8248</strong></td><td>0.8214</td><td>0.8201</td><td>0.8165</td></tr><tr><td>A549</td><td><strong>0.8154</strong></td><td>0.8152</td><td></td><td>0.8132</td><td>0.8137</td><td>0.8153</td><td>0.8103</td></tr><tr><td>BT20</td><td>0.7758</td><td>0.8187</td><td>0.8173</td><td>0.8164</td><td><strong>0.8267</strong></td><td>0.8225</td><td>0.8102</td></tr><tr><td>HA1E</td><td>0.8384</td><td><strong>0.8403</strong></td><td>0.8380</td><td>0.8397</td><td>0.8382</td><td>0.8400</td><td>0.8342</td></tr><tr><td>HCC515</td><td>0.8420</td><td>0.8444</td><td>0.8440</td><td>0.8449</td><td>0.8448</td><td><strong>0.8473</strong></td><td>0.8423</td></tr><tr><td>HEK293T</td><td>0.8383</td><td>0.8835</td><td>0.8797</td><td>0.8766</td><td>0.8838</td><td><strong>0.8864</strong></td><td>0.8739</td></tr><tr><td>HEPG2</td><td>0.8341</td><td>0.8635</td><td>0.8638</td><td>0.8636</td><td><strong>0.8645</strong></td><td>0.8600</td><td>0.8507</td></tr><tr><td>HL60</td><td>0.7917</td><td><strong>0.8925</strong></td><td>0.8744</td><td>0.8881</td><td>0.8737</td><td>0.8739</td><td>0.8803</td></tr><tr><td>HS578T</td><td>0.7843</td><td>0.8080</td><td><strong>0.8082</strong></td><td>0.8037</td><td>0.8063</td><td>0.8077</td><td>0.7993</td></tr><tr><td>HT29</td><td>0.7974</td><td><strong>0.8032</strong></td><td>0.7954</td><td></td><td>0.7932</td><td>0.7960</td><td>0.7915</td></tr><tr><td>HUH7</td><td>0.7682</td><td>0.7954</td><td>0.7953</td><td><strong>0.7988</strong></td><td>0.7963</td><td>0.7957</td><td>0.7816</td></tr><tr><td>JURKAT</td><td>0.7917</td><td><strong>0.8781</strong></td><td>0.8506</td><td>0.8701</td><td>0.8591</td><td>0.8714</td><td>0.8422</td></tr><tr><td>MCF10A</td><td>0.7955</td><td>0.8200</td><td>0.8192</td><td>0.8196</td><td><strong>0.8215</strong></td><td>0.8165</td><td>0.8178</td></tr><tr><td>MCF7</td><td><strong>0.8430</strong></td><td>0.8425</td><td>0.8415</td><td>0.8414</td><td></td><td>0.8430</td><td>0.8395</td></tr><tr><td>MDAMB231</td><td>0.7953</td><td>0.8395</td><td>0.8367</td><td>0.8345</td><td><strong>0.8431</strong></td><td>0.8397</td><td>0.8335</td></tr><tr><td>NKDBA</td><td>0.7241</td><td>0.7948</td><td>0.7983</td><td>0.7914</td><td><strong>0.8002</strong></td><td>0.7967</td><td>0.7926</td></tr><tr><td>NOMO1</td><td>0.7934</td><td>0.8044</td><td>0.7665</td><td>0.7420</td><td><strong>0.8467</strong></td><td>0.8044</td><td>0.6887</td></tr><tr><td>PC3</td><td>0.8327</td><td>0.8322</td><td>0.8323</td><td>0.8314</td><td><strong>0.8345</strong></td><td></td><td>0.8320</td></tr><tr><td>SKBR3</td><td>0.7951</td><td>0.8233</td><td>0.8225</td><td>0.8255</td><td><strong>0.8292</strong></td><td>0.8203</td><td>0.8208</td></tr><tr><td>THP1</td><td>0.7590</td><td>0.7945</td><td>0.8156</td><td>0.7439</td><td><strong>0.8399</strong></td><td>0.7845</td><td>0.7082</td></tr><tr><td>U266</td><td>0.7765</td><td>0.8192</td><td>0.8076</td><td>0.8217</td><td>0.8344</td><td>0.8241</td><td><strong>0.8345</strong></td></tr><tr><td>U937</td><td>0.7569</td><td><strong>0.8050</strong></td><td>0.7642</td><td>0.8025</td><td>0.7370</td><td>0.7168</td><td>0.7705</td></tr><tr><td>VCAP</td><td>0.8564</td><td>0.8554</td><td>0.8541</td><td>0.8539</td><td><strong>0.8570</strong></td><td>0.8561</td><td></td></tr></tbody></table>
for major changes in the network structure. Upon further investigation, we de-
duced that the almost static but low error rate was limited by the data itself.
While producing impressive results, the neural network itself wasn't modeling
the data correctly, and instead was basically training on the test data, due to
the repetitions in the dataset. This was also the reason behind the change in the
layer sizes since smaller layers could produce similar results faster.

To eliminate the compound repetition, we’ve introduced the cold-drug-split.
Repeating the original methodology with this new split resulted in 23.05% lower
AUC scores on average. These results can be seen on the No-TL columns of
the Table 3. The most improvement for the up-regulated models was on the
JURKAT cell line when A375 cell line data was used as the source; which boosted
the AUC score by 22.81%. For down-regulated, it was on the HCC515 cell line
by 14.38% when the PC3 was used as the source. The results of transfer learning
experiments on cold-drug-split can be seen on Table 3. As mentioned before, we
regard cold-drug-split as the most realistic method of the three. Experiments on
this split resulted in average improvements of 9.70% and 8.29% on AUC scores
for up and down-regulated models, respectively.

The second splitting method we introduced, transfer-drug-split, while a bit
unrealistic compared to cold-drug-split, posed an important challenge to the
model training. We’ve re-tested the models we trained in cold-drug-split by re-
moving the compounds that also were tested on the source cell line from the
target test splits, producing a testing environment in which the model was tested
purely tested on its ability to process new compounds. On transfer-drug-split,
the AUC scores on average improved by 1.42% and -0.73% for up and down-
regulated models. However, the maximum AUC gains were comparable to the
ones on the other splits. For up-regulated, training the JURKAT cell line on top
of the A375 resulted in 19.88% improved AUC. Among down-regulated it was
HS578T which improved its AUC score by 6.74% when trained after MCF7.

Detailed AUC scores of transfer-drug-split experiments can be seen on Ta-
ble 4, for up and down-regulated model results.

3 Discussion

3.1 Best Improvements

When we look at the top 10 most improved AUC scores in Table 5, several trends emerge that we can comment on.

Firstly, every experiment on the table is an up differential expression (DE)
model. If we were to take an average of every up, and down differential expres-
sion experiments separately, we can see the up regulated experiments are more
effected than down regulated ones. This can be correlated to the greater up
regulated sample counts compared to the downs.

Another visible trend is in the splits. Other than one exception, every split in
the table is either from a cold-drug-split or a transfer-drug-split experiment. This
is mainly because of the 23% decrease in the AUC values when we'd switched
<table><caption>**Table 3.** AUC scores of regulated genes on cold-drug-split 10-fold data, rows and columns indicate target and source cell lines. The best results in a row are indicated in bold.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Target Cell ↓</th><th colspan="7">Source Cell</th></tr><tr><th>No-TL</th><th>A375</th><th>A549</th><th>HT29</th><th>MCF7</th><th>PC3</th><th>VCAP</th></tr></thead><tbody><tr><td rowspan="24">Up</td><td>A375</td><td>0.6145</td><td></td><td>0.6633</td><td>0.6499</td><td>0.6558</td><td><strong>0.6652</strong></td><td>0.6304</td></tr><tr><td>A549</td><td>0.5864</td><td>0.5944</td><td></td><td>0.5851</td><td>0.6294</td><td><strong>0.6445</strong></td><td>0.6145</td></tr><tr><td>BT20</td><td>0.5952</td><td>0.5460</td><td>0.6316</td><td>0.5801</td><td><strong>0.6553</strong></td><td>0.6268</td><td>0.6013</td></tr><tr><td>HA1E</td><td>0.6220</td><td>0.6601</td><td>0.6701</td><td>0.6482</td><td>0.6627</td><td><strong>0.6859</strong></td><td>0.6666</td></tr><tr><td>HCC515</td><td>0.6136</td><td>0.6401</td><td>0.6761</td><td>0.6507</td><td>0.6679</td><td><strong>0.6970</strong></td><td>0.6664</td></tr><tr><td>HEK293T</td><td>0.6165</td><td><strong>0.6971</strong></td><td>0.6238</td><td>0.6698</td><td>0.6543</td><td>0.6348</td><td>0.6466</td></tr><tr><td>HEPG2</td><td>0.6096</td><td>0.7297</td><td>0.7192</td><td>0.6761</td><td>0.6893</td><td><strong>0.7388</strong></td><td>0.6224</td></tr><tr><td>HL60</td><td>0.6063</td><td>0.6183</td><td>0.5397</td><td><strong>0.6665</strong></td><td>0.5670</td><td>0.5671</td><td>0.5604</td></tr><tr><td>HS578T</td><td>0.5762</td><td>0.5796</td><td>0.6509</td><td>0.6306</td><td><strong>0.6725</strong></td><td>0.6436</td><td>0.6307</td></tr><tr><td>HT29</td><td>0.6197</td><td><strong>0.6892</strong></td><td>0.6756</td><td></td><td>0.6742</td><td>0.6719</td><td>0.6374</td></tr><tr><td>HUH7</td><td>0.6428</td><td>0.6197</td><td>0.6856</td><td>0.6488</td><td>0.6749</td><td><strong>0.6980</strong></td><td>0.6679</td></tr><tr><td>JURKAT</td><td>0.5558</td><td><strong>0.6826</strong></td><td>0.5527</td><td>0.5816</td><td>0.5898</td><td>0.6307</td><td>0.6205</td></tr><tr><td>MCF10A</td><td>0.6434</td><td>0.6131</td><td>0.6360</td><td>0.6291</td><td><strong>0.6594</strong></td><td>0.6460</td><td>0.6323</td></tr><tr><td>MCF7</td><td>0.6059</td><td>0.6151</td><td>0.6569</td><td>0.6100</td><td></td><td><strong>0.6752</strong></td><td>0.6477</td></tr><tr><td>MDAMB231</td><td>0.6191</td><td>0.5859</td><td>0.6300</td><td>0.6138</td><td><strong>0.6896</strong></td><td>0.6586</td><td>0.6288</td></tr><tr><td>NKDBA</td><td>0.6121</td><td>0.5950</td><td>0.6267</td><td>0.6130</td><td><strong>0.6647</strong></td><td>0.6091</td><td>0.6129</td></tr><tr><td>NOMO1</td><td>0.5888</td><td>0.5100</td><td>0.5758</td><td>0.5689</td><td>0.5590</td><td><strong>0.5920</strong></td><td>0.4933</td></tr><tr><td>PC3</td><td>0.6055</td><td>0.6153</td><td>0.6453</td><td>0.6074</td><td><strong>0.6644</strong></td><td></td><td>0.6386</td></tr><tr><td>SKBR3</td><td>0.6057</td><td>0.5599</td><td>0.6161</td><td>0.5757</td><td><strong>0.6575</strong></td><td>0.6311</td><td>0.5835</td></tr><tr><td>THP1</td><td>0.5849</td><td>0.5427</td><td>0.5922</td><td>0.5841</td><td><strong>0.6019</strong></td><td>0.5848</td><td>0.5697</td></tr><tr><td>U266</td><td><strong>0.6430</strong></td><td>0.6557</td><td>0.6408</td><td>0.6136</td><td>0.6035</td><td>0.6249</td><td>0.5553</td></tr><tr><td>U937</td><td>0.5908</td><td>0.5800</td><td>0.6002</td><td><strong>0.6067</strong></td><td>0.5710</td><td>0.6046</td><td>0.5772</td></tr><tr><td>VCAP</td><td>0.5956</td><td>0.5970</td><td>0.6238</td><td>0.5924</td><td>0.6281</td><td><strong>0.6387</strong></td><td></td></tr></tbody></table>

<table><tbody><tr><td rowspan="24">Down</td><td>A375</td><td>0.6228</td><td></td><td>0.6648</td><td>0.6575</td><td>0.6563</td><td><strong>0.6671</strong></td><td>0.6317</td></tr><tr><td>A549</td><td>0.5799</td><td>0.5884</td><td></td><td>0.5933</td><td>0.6309</td><td><strong>0.6352</strong></td><td>0.6162</td></tr><tr><td>BT20</td><td>0.6236</td><td>0.6222</td><td>0.6620</td><td>0.6303</td><td><strong>0.7064</strong></td><td>0.6697</td><td>0.6188</td></tr><tr><td>HA1E</td><td>0.6331</td><td>0.6685</td><td>0.6673</td><td>0.6505</td><td>0.6693</td><td><strong>0.6954</strong></td><td>0.6723</td></tr><tr><td>HCC515</td><td>0.6184</td><td>0.6440</td><td>0.6836</td><td>0.6488</td><td>0.6885</td><td><strong>0.7073</strong></td><td>0.6776</td></tr><tr><td>HEK293T</td><td>0.6982</td><td>0.7408</td><td>0.7752</td><td>0.7292</td><td><strong>0.7781</strong></td><td>0.6995</td><td>0.7610</td></tr><tr><td>HEPG2</td><td>0.6653</td><td><strong>0.7457</strong></td><td>0.7450</td><td>0.7039</td><td>0.7415</td><td>0.7421</td><td>0.6554</td></tr><tr><td>HL60</td><td><strong>0.7314</strong></td><td>0.6375</td><td>0.6507</td><td>0.6811</td><td>0.6235</td><td>0.6327</td><td>0.6659</td></tr><tr><td>HS578T</td><td>0.6121</td><td>0.6203</td><td>0.6621</td><td>0.6469</td><td><strong>0.6964</strong></td><td>0.6575</td><td>0.6374</td></tr><tr><td>HT29</td><td>0.6199</td><td><strong>0.6969</strong></td><td>0.6709</td><td></td><td>0.6742</td><td>0.6718</td><td>0.6396</td></tr><tr><td>HUH7</td><td>0.6469</td><td>0.6543</td><td>0.6766</td><td>0.6777</td><td><strong>0.6955</strong></td><td>0.6708</td><td>0.6783</td></tr><tr><td>JURKAT</td><td>0.6679</td><td><strong>0.7213</strong></td><td>0.6766</td><td>0.6820</td><td>0.5887</td><td>0.6649</td><td>0.6359</td></tr><tr><td>MCF10A</td><td>0.6355</td><td>0.6396</td><td>0.6510</td><td>0.6559</td><td><strong>0.6805</strong></td><td>0.6800</td><td>0.6469</td></tr><tr><td>MCF7</td><td>0.6148</td><td>0.6240</td><td>0.6600</td><td>0.6207</td><td></td><td><strong>0.6950</strong></td><td>0.6575</td></tr><tr><td>MDAMB231</td><td>0.6595</td><td>0.6323</td><td>0.6702</td><td>0.6395</td><td><strong>0.7080</strong></td><td>0.6845</td><td>0.6370</td></tr><tr><td>NKDBA</td><td>0.6320</td><td>0.6387</td><td>0.6364</td><td>0.6289</td><td><strong>0.6673</strong></td><td>0.6345</td><td>0.6252</td></tr><tr><td>NOMO1</td><td>0.5725</td><td>0.5049</td><td>0.5858</td><td>0.5596</td><td>0.5649</td><td><strong>0.6268</strong></td><td>0.5099</td></tr><tr><td>PC3</td><td>0.6118</td><td>0.6166</td><td>0.6526</td><td>0.6209</td><td><strong>0.6740</strong></td><td></td><td>0.6489</td></tr><tr><td>SKBR3</td><td>0.6472</td><td>0.6325</td><td>0.6320</td><td>0.6474</td><td><strong>0.6937</strong></td><td>0.6545</td><td>0.6116</td></tr><tr><td>THP1</td><td>0.6326</td><td>0.5687</td><td>0.5944</td><td>0.5802</td><td><strong>0.6413</strong></td><td>0.6338</td><td>0.5806</td></tr><tr><td>U266</td><td>0.6799</td><td>0.6907</td><td>0.7053</td><td>0.6524</td><td>0.6506</td><td><strong>0.7065</strong></td><td>0.5438</td></tr><tr><td>U937</td><td>0.5641</td><td>0.5860</td><td>0.5903</td><td>0.5786</td><td>0.5697</td><td>0.5750</td><td><strong>0.5913</strong></td></tr><tr><td>VCAP</td><td>0.5965</td><td>0.6014</td><td>0.6197</td><td>0.6007</td><td>0.6272</td><td><strong>0.6458</strong></td><td></td></tr></tbody></table>
<table><caption>**Table 4.** AUC scores of regulated genes on transfer-drug-split 10-fold data, rows and columns indicate target and source cell lines. The best results in a row are indicated in bold.</caption><thead><tr><th rowspan="2">Target Cell ↓</th><th rowspan="2">No-TL</th><th colspan="6">Source Cell</th></tr><tr><th>A375</th><th>A549</th><th>HT29</th><th>MCF7</th><th>PC3</th><th>VCAP</th></tr></thead><tbody><tr><td>A375</td><td><strong>0.6145</strong></td><td></td><td>0.6065</td><td>0.5943</td><td>0.5933</td><td>0.6060</td><td>0.5767</td></tr><tr><td>A549</td><td><strong>0.5864</strong></td><td>0.5414</td><td></td><td>0.5354</td><td>0.5711</td><td>0.5780</td><td>0.5572</td></tr><tr><td>BT20</td><td>0.5952</td><td>0.5156</td><td>0.5765</td><td>0.5440</td><td><strong>0.6019</strong></td><td>0.5729</td><td>0.5409</td></tr><tr><td>HA1E</td><td>0.6220</td><td>0.5979</td><td>0.6096</td><td>0.5913</td><td>0.6059</td><td><strong>0.6252</strong></td><td>0.6085</td></tr><tr><td>HCC515</td><td>0.6136</td><td>0.5843</td><td>0.6164</td><td>0.5899</td><td>0.6155</td><td><strong>0.6381</strong></td><td>0.6169</td></tr><tr><td>HEK293T</td><td>0.6165</td><td><strong>0.6302</strong></td><td>0.5973</td><td>0.5891</td><td>0.5685</td><td>0.5517</td><td>0.5673</td></tr><tr><td>HEPG2</td><td>0.6096</td><td>0.6504</td><td>0.6458</td><td>0.6255</td><td>0.6302</td><td><strong>0.6517</strong></td><td>0.5811</td></tr><tr><td>HL60</td><td>0.6063</td><td>0.5978</td><td>0.5479</td><td><strong>0.6268</strong></td><td>0.5572</td><td>0.5543</td><td>0.5590</td></tr><tr><td>HS578T</td><td>0.5762</td><td>0.5426</td><td>0.5796</td><td>0.5746</td><td><strong>0.6263</strong></td><td>0.5851</td><td>0.5655</td></tr><tr><td>HT29</td><td>0.6197</td><td><strong>0.6223</strong></td><td>0.6076</td><td></td><td>0.6028</td><td>0.5999</td><td>0.5781</td></tr><tr><td>HUH7</td><td><strong>0.6428</strong></td><td>0.5984</td><td>0.6334</td><td>0.6157</td><td>0.6274</td><td>0.6221</td><td>0.5978</td></tr><tr><td>JURKAT</td><td>0.5558</td><td><strong>0.6663</strong></td><td>0.6485</td><td>0.6325</td><td>0.6208</td><td>0.6652</td><td>0.6300</td></tr><tr><td>MCF10A</td><td><strong>0.6434</strong></td><td>0.5687</td><td>0.5845</td><td>0.5827</td><td>0.6160</td><td>0.6005</td><td>0.5722</td></tr><tr><td>MCF7</td><td>0.6059</td><td>0.5612</td><td>0.5896</td><td>0.5569</td><td></td><td><strong>0.6139</strong></td><td>0.5857</td></tr><tr><td>MDAMB231</td><td>0.6191</td><td>0.5545</td><td>0.5905</td><td>0.5852</td><td><strong>0.6397</strong></td><td>0.6074</td><td>0.5700</td></tr><tr><td>NKDBA</td><td>0.6121</td><td>0.5408</td><td>0.5623</td><td>0.5565</td><td><strong>0.6205</strong></td><td>0.5665</td><td>0.5797</td></tr><tr><td>NOMO1</td><td><strong>0.5888</strong></td><td>0.5472</td><td>0.4782</td><td>0.4862</td><td>0.4978</td><td>0.5132</td><td>0.4282</td></tr><tr><td>PC3</td><td>0.6055</td><td>0.5619</td><td>0.5874</td><td>0.5554</td><td><strong>0.6061</strong></td><td></td><td>0.5835</td></tr><tr><td>SKBR3</td><td>0.6057</td><td>0.5306</td><td>0.5651</td><td>0.5460</td><td><strong>0.6196</strong></td><td>0.5798</td><td>0.5416</td></tr><tr><td>THP1</td><td><strong>0.5849</strong></td><td>0.5095</td><td>0.5489</td><td>0.5360</td><td>0.5417</td><td>0.5522</td><td>0.4967</td></tr><tr><td>U266</td><td>0.6430</td><td><strong>0.6619</strong></td><td>0.6252</td><td>0.5920</td><td>0.5915</td><td>0.6056</td><td>0.5308</td></tr><tr><td>U937</td><td><strong>0.5968</strong></td><td>0.5570</td><td>0.5632</td><td>0.5792</td><td>0.5665</td><td>0.5739</td><td>0.5360</td></tr><tr><td>VCAP</td><td><strong>0.5956</strong></td><td>0.5446</td><td>0.5680</td><td>0.5419</td><td>0.5755</td><td>0.5845</td><td></td></tr></tbody></table>

<table><tbody><tr><td>A375</td><td><strong>0.6228</strong></td><td></td><td>0.6085</td><td>0.6005</td><td>0.5962</td><td>0.6085</td><td>0.5765</td></tr><tr><td>A549</td><td><strong>0.5799</strong></td><td>0.5376</td><td></td><td>0.5403</td><td>0.5727</td><td>0.5719</td><td>0.5594</td></tr><tr><td>BT20</td><td>0.6236</td><td>0.5764</td><td>0.5992</td><td>0.5851</td><td><strong>0.6353</strong></td><td>0.6149</td><td>0.5696</td></tr><tr><td>HA1E</td><td>0.6331</td><td>0.6029</td><td>0.6081</td><td>0.5890</td><td>0.6085</td><td><strong>0.6337</strong></td><td>0.6155</td></tr><tr><td>HCC515</td><td>0.6184</td><td>0.5854</td><td>0.6166</td><td>0.5870</td><td>0.6239</td><td><strong>0.6455</strong></td><td>0.6270</td></tr><tr><td>HEK293T</td><td><strong>0.6982</strong></td><td>0.6496</td><td>0.5940</td><td>0.5941</td><td>0.6015</td><td>0.6014</td><td>0.6125</td></tr><tr><td>HEPG2</td><td>0.6653</td><td><strong>0.6693</strong></td><td>0.6693</td><td>0.6476</td><td>0.6639</td><td>0.6611</td><td>0.6072</td></tr><tr><td>HL60</td><td><strong>0.7314</strong></td><td>0.6355</td><td>0.6215</td><td>0.6722</td><td>0.6274</td><td>0.6164</td><td>0.6266</td></tr><tr><td>HS578T</td><td>0.6121</td><td>0.5725</td><td>0.6025</td><td>0.5900</td><td><strong>0.6534</strong></td><td>0.6029</td><td>0.5771</td></tr><tr><td>HT29</td><td>0.6199</td><td><strong>0.6274</strong></td><td>0.6050</td><td></td><td>0.6046</td><td>0.6039</td><td>0.5761</td></tr><tr><td>HUH7</td><td><strong>0.6469</strong></td><td>0.6113</td><td>0.6150</td><td>0.6275</td><td>0.6397</td><td>0.6126</td><td>0.6026</td></tr><tr><td>JURKAT</td><td>0.7107</td><td><strong>0.7162</strong></td><td>0.6795</td><td>0.6728</td><td>0.6581</td><td>0.6953</td><td>0.6088</td></tr><tr><td>MCF10A</td><td><strong>0.6355</strong></td><td>0.5980</td><td>0.6067</td><td>0.6163</td><td>0.6281</td><td>0.6200</td><td>0.6011</td></tr><tr><td>MCF7</td><td>0.6148</td><td>0.5686</td><td>0.5962</td><td>0.5646</td><td></td><td><strong>0.6200</strong></td><td>0.5938</td></tr><tr><td>MDAMB231</td><td>0.6595</td><td>0.5882</td><td>0.6082</td><td>0.6025</td><td><strong>0.6615</strong></td><td>0.6188</td><td>0.5855</td></tr><tr><td>NKDBA</td><td><strong>0.6320</strong></td><td>0.5843</td><td>0.5739</td><td>0.5785</td><td>0.6219</td><td>0.5668</td><td>0.5589</td></tr><tr><td>NOMO1</td><td><strong>0.5725</strong></td><td>0.4693</td><td>0.5309</td><td>0.4847</td><td>0.5083</td><td>0.5448</td><td>0.4489</td></tr><tr><td>PC3</td><td>0.6118</td><td>0.5634</td><td>0.5925</td><td>0.5655</td><td><strong>0.6156</strong></td><td></td><td>0.5940</td></tr><tr><td>SKBR3</td><td>0.6472</td><td>0.5858</td><td>0.5971</td><td>0.6041</td><td><strong>0.6495</strong></td><td>0.6131</td><td>0.5767</td></tr><tr><td>THP1</td><td><strong>0.6326</strong></td><td>0.5500</td><td>0.5382</td><td>0.5443</td><td>0.5967</td><td>0.5946</td><td>0.5165</td></tr><tr><td>U266</td><td><strong>0.6799</strong></td><td>0.6535</td><td>0.6438</td><td>0.5995</td><td>0.6150</td><td>0.6452</td><td>0.5164</td></tr><tr><td>U937</td><td>0.5641</td><td>0.5605</td><td>0.5575</td><td>0.5747</td><td><strong>0.5837</strong></td><td>0.5476</td><td>0.5779</td></tr><tr><td>VCAP</td><td><strong>0.5965</strong></td><td>0.5462</td><td>0.5637</td><td>0.5464</td><td>0.5753</td><td>0.5888</td><td></td></tr></tbody></table>
to cold-drug-split. It is easier to improve upon a worse baseline. Also, the complications in the random-drug-split makes it harder to improve its scores, of which we will discuss in the next section.

Lastly, two target cell lines dominate the top 10: JURKAT, and HEPG2. Interestingly, the reason behind them are polar opposites. For HEPG2, similar improvements are not visible in the transfer-drug-split experiments, therefore the benefits of transfer learning must be coming from the same drugs that are also tested on the source cell line. For the JURKAT cell line, the average improvement goes up by 63% when we switch to transfer-drug-split, which eliminates drugs tested on the source cell line. This points to better generalization for the target, and greater transferred knowledge from the source cell line.

<table><caption>**Table 5.** Transfer learning experiments that showed the most improvements for the direction of differential expression (DE) and split.</caption><thead><tr><th></th><th>Source - Target</th><th>Improvement (DE, split)</th></tr></thead><tbody><tr><td>1</td><td>A375 - JURKAT</td><td>%22.81 (Up, cold)</td></tr><tr><td>2</td><td>PC3 - HEPG2</td><td>%21.19 (Up, cold)</td></tr><tr><td>3</td><td>A375 - JURKAT</td><td>%19.88 (Up, transfer)</td></tr><tr><td>4</td><td>A375 - HEPG2</td><td>%19.70 (Up, cold)</td></tr><tr><td>5</td><td>A549 - JURKAT</td><td>%19.69 (Up, transfer)</td></tr><tr><td>6</td><td>PC3 - JURKAT</td><td>%19.68 (Up, transfer)</td></tr><tr><td>7</td><td>A549 - HEPG2</td><td>%17.98 (Up, cold)</td></tr><tr><td>8</td><td>MCF7 - HS578T</td><td>%16.71 (Up, cold)</td></tr><tr><td>9</td><td>A549 - JURKAT</td><td>%16.69 (Up, transfer)</td></tr><tr><td>10</td><td>MCF7 - NKDBA</td><td>%15.73 (Up, random)</td></tr></tbody></table>

## 3.2 Complications of Using Random Split

The main idea behind DeepCOP is simple, yet effective: Utilizing differentiable characteristics of outputs lets us treat multiple genes' perturbation values as a single objective. This method populates the amount of data for each compound, retains the relation between individual genes, and gives each output a new meaning. However, for this theory to simulate real-life occurrences, the population should be done after the required validation split. To further demonstrate as an example; the THP1 cell line included in this study contains 18 different compounds. From the original data, we get 978 gene perturbations for each compound. After populating this data, we end up with 17,604 samples. However, we only have 18 unique compounds and 978 unique gene descriptors. Splitting this dataset into 10 random folds almost guarantees the possibility of data leakage.

There are two causes for this data leakage; duplicate genes, and compounds. In the L1000 dataset, an experiment corresponds to one compound, and its gene perturbation effects on a cell line. After the data processing step, we have ended up with the same compound repeated in 978 samples to predict each gene
separately. Randomly splitting this data means separating the gene outputs of a
single experiment into different folds. It also implies that our model is learning
to predict individual gene perturbations from the outcomes of other genes, for
the same compound. This is an unrealistic and trivial task, since every gene
perturbation is measured simultaneously for that experiment.

To counteract the data leakage caused by gene repetition, we would have to
implement a transfer-gene split, where we would split 978 genes into different
folds and populate the folds with the compounds. However, this splitting method
is the polar opposite of the objective of HTS, since we're trying to simulate an
untested compound's effects on existing genes and not the other way around.
Therefore, can be ignored for this study.

3.3 Random-split in Source Model Training

In this study, we have trained our source models on a random 95%-5% train-
validation split with early stopping. From our previous remarks about random-
split, the credibility of these source models should also be questioned.

For our initial experiments, we were using the non-TL models we've trained
for the random-split experiments as source models. When the problem of random-
split arose, we've switched our source models to use 100% of the source data.
This resulted in worse evaluations for the target model, due to the source net-
work not being optimized for such a task. In an effort to minimize the negative
effects of data leakage, while keeping the benefits of early stopping; we've de-
cided on the 95%-5% random split. It should be noted that the 95%-5% split still
negatively affects the training of the source models, and for a state-of-the-art
product; source models should be first optimized on a compound-based split and
trained on 100% of the source data.

3.4 Splitting Up and Down Regulations

In L1000, gene perturbations can be split into 3 categories; up-regulated, neutral, and down-regulated. Multi-class classification is a special case of classification problems and is usually harder to model compared to binary classification problems.

DeepCOP and subsequently this study divides what normally is a multi-class classification problem into 2 binary classifications. While this widely-used practice makes the data easier to model, it also comes at a cost of lost knowledge.

Specifically, in this application, the relation between up and down-regulated
outputs is lost, because our binary classifications are “actively up-regulated to
not significantly up-regulated” and “actively down-regulated to not significantly
down-regulated”. Training an ideal for the first problem would require a greater
penalty to classify an actively down-regulated sample as actively up-regulated
than to classify a non-regulated sample as one; and vice versa for the second
problem.

Modeling the problem as multi-class showed that the features are not descrip-
tive enough to model the added difficulty, at least for this type of neural network.
The most obvious solution would be using a custom loss function that uses the
results of an oppositely-regulated problem. Another possible solution could be
transfer learning. Utilizing the weights of up-regulated models in down-regulated
models, or modeling up and down-regulation as subnetworks of the same neural
network, has the potential to utilize the lost knowledge.

4 Conclusion

In this study, we tried to exhibit the benefit of using the natural genetic sim-
ilarities in between cell lines for chemogenomic machine learning. While doing
so, we have introduced 2 splitting methods to DeepCOP to better evaluate the
strength of its methodology. Based upon the initial study of DeepCOP, our
results have shown improvements of 10.65% on average for the introduced cold-
drug-split. While there is a definite improvement in the results, this study should
be treated as a simple demonstration, and can be further optimized for even
better results. As it stands, this study is a testament for the usage of transfer
learning in chemogenomics, and a demonstration on how it can be utilized. Mov-
ing forward, we're planning on implementing other methods of transfer learning,
applying transfer learning on more than 2 cell line combinations, and optimize
them to further illustrate the optimum possible result out of this methodology.

References

1. Bredel, M., Jacoby, E.: Chemogenomics: an emerging strategy for rapid target and drug discovery. Nature Reviews Genetics **5**(4), 262–275 (2004)

2. Consortium, G.O.: The gene ontology (go) database and informatics resource. Nucleic acids research **32**(suppl_1), D258–D261 (2004)

3. Gobbi, A., Poppinger, D.: Genetic optimization of combinatorial libraries. Biotechnology and bioengineering **61**(1), 47–54 (1998)

4. Li, H., Tian, S., Li, Y., Fang, Q., Tan, R., Pan, Y., Huang, C., Xu, Y., Gao, X.: Modern deep learning in bioinformatics. Journal of molecular cell biology **12**(11), 823–827 (2020)

5. Li, Y., Huang, C., Ding, L., Li, Z., Pan, Y., Gao, X.: Deep learning in bioinformatics: Introduction, application, and perspective in the big data era. Methods **166**, 4–21 (2019)

6. RDKit: Open-source cheminformatics. http://www.rdkit.org, [Online; Release 2021.3]

7. Rogers, D., Hahn, M.: Extended-connectivity fingerprints. Journal of chemical information and modeling **50**(5), 742–754 (2010)

8. Subramanian, A., Narayan, R., Corsello, S.M., Peck, D.D., Natoli, T.E., Lu, X., Gould, J., Davis, J.F., Tubelli, A.A., Asiedu, J.K., et al.: A next generation connectivity map: L1000 platform and the first 1,000,000 profiles. Cell **171**(6), 1437–1452 (2017)

9. Tang, B., Pan, Z., Yin, K., Khateeb, A.: Recent advances of deep learning in bioinformatics and computational biology. Frontiers in genetics **10**, 214 (2019)

10. Woo, G., Fernandez, M., Hsing, M., Lack, N.A., Cavga, A.D., Cherkasov, A.: Deepcop: deep learning-based approach to predict gene regulating effects of small molecules. Bioinformatics **36**(3), 813–818 (2020)